It is either a JavaScript object literal or more specifically JSON when it comes to sending parameters over Ajax. JSON is subset of JavaScript object literals.
For example:
// This is JSON data sent via the Ajax request (JSON is subset of JavaScript object literals)
var json = {id: 1, first_name: "John", last_name: "Smith"};
// This is a JavaScript object literal, it is not used for transfer of data so doesn't need to be JSON
var jsol = {type: 'POST', url: url, data: json};
$.ajax(jsol);
In this case you're passing an object containing your settings to the plugin. The plugin can deal with this as a object, whatever it's referenced as, for example:
settings.option1 //the option you passed in.
Of course it has a lot more uses, but this is the most common example in jQuery. The same is true for the .animate(), $.ajax(), .css() functions, etc. Anything that takes properties generally uses this format.
As requested, some other examples:
Any object inside the passed object can be a function as well, not only properties, for example:
This would set the focus event of that input to have an alert. Another is extending an object, adding properties to it, like this:
var person = { first_name: "John" };
$.extend(person, { last_name: "Smith" });
//equivalent to:
person.last_name = "Smith";
//or:
person["last_name"] = "Smith";
Now person has the last_name property. This is often used by plugins as well, to take the default settings, then merge any settings you passed in, overwriting with any settings you specified, using defaults for the rest.
Why are we using it? Well...that's how JavaScript works, and in the jQuery spirit: it's an extremely terse and flexible way to pass information.
In this context, the parentheses "(...)" following an expression, such as $.ajaxSetup, causes the function specified by the expression to be called.
The expression inside the parentheses (which could be a comma separated list of expressions) results in a value (or a list of values) that is the argument(s) passed to the function.
Finally, when "{...}" is used in an expression context, it constructs an object with the name-value properties specified. This is like JSON but it is, more generally, any legal JS object literal.