What kind of system do you use (or smart people) use for a database of records that are used for distributed configuration settings?

A key value store would work but I hear those are normally used for caching and not as an updatable system for configuration settings.

I asked this question on StackOverflow and the overwhelming response was to use a SQL database.

SQL database is also fine but that isn't the role for a SQL database.

...
Imagine:

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200

.ini

> SQL database is also fine but that isn't the role for a SQL database.

Isn't it?  What makes it unsuitable?  You haven't really provided enough information about what you're going to store, how many records we're talking about, or even why it's distributed.

If your configuration is well structured, wouldn't that map exactly to a database table (columns as settings, values as selections).

Why not a relational DB?

Distributed config info?

Another vote for INI file.

(as long as there's no hierarchical information stored)

In a web environment.

Still not enough information.  Is there some reason why you wouldn't want to go SQL DB?  Does this project not use a DB at all?

"Is there some reason why you wouldn't want to go SQL DB? "

It does.

A SQL DB is normally used for relational application data.

Is there something web/distributed exclusively for updating configuration settings.

With a DB, I have to go through the work of connecting to the database, querying my data.

I was hoping in 2012 there was an online system used for updating configuration settings.  Something that is fast, persistent and allows for periodic updates.

...

Ironically, distributed key stores like MongoDB and Cassandra are used for caching solutions not really for config settings.

"Is there some reason why you wouldn't want to go SQL DB? "

Nothing really, I was just curious if something else existed.

Why not Zoidberg?

MongoDB is a document store. Both Cassandra and MongoDB are persistent and are seldom used for caching. Caching generally uses things like Redis or Memcache/Membase.

It sounds like you're searching for a solution to a problem you also haven't discovered. For your basic description as a simple shared file sounds overwhelming fine.

> A SQL DB is normally used for relational application data.

That might be over thinking it.  If you can structure it into a table, then a SQL DB is a reasonable choice.  The data doesn't really have to be "relational".  There's lot of data that doesn't fit well into a table -- at that point a SQL DB might be a poor choice.

> Is there something web/distributed exclusively for updating configuration settings.

Seems a bit specific.  You could use any of the NoSQL solutions that include persistence. 

> With a DB, I have to go through the work of connecting to the database, querying my data.

Isn't that true of any client/server solution.  If I'm using, for example, memcache as a caching layer I still need to connect to it and query the data.

Seems you want something too specific to actually exist.

This is a simple property file.  Normally in the Java world, you may include these settings are part of your web app deployment.

So you deploy your property settings as:


MyWebAppOnTheFileSystemAsReadOnly/MySettings.properties:

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200

This is fine but those settings are read-only and can only get updated when you redeploy your web application.

...

So now you move the settings to a random location on the filesystem:


/usr/local/var/MySettings.properties

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200

...
This approach makes it easier to update the settings but you have set it up so only an admin will have to edit the file manually using vi or some unix editor.

The application developer or administrator can't update the configuration settings online.

...

So then you move your application to database

MyConfigTable@MYDB
user=confmaster
pwd=configuration

select * from MyConfigTable

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200


...

That approach also works but you are using a relational database for simple key value system.

Does anyone here actually work in enterprise development?

"Isn't that true of any client/server solution. "

True, but you avoid the time for authentication and the database lookup, etc.

NoSQL can be faster than a large database.

If the app uses DB already, then yes. If not, it's an overkill. +1 for ini file

Wayne,

What NoSQL solutions are people using for this type of task.

MongoDB, MemcacheDB, CouchDB, Cassandra?

And J2EE has Jndi.

"NoSQL can be faster than a large database."

Ugh.

Step back, everyone, Bot is doing "enterprise" work here.

Ha ha ha.

"For your basic description as a simple shared file "

It was a response to your comment.  A simple shared file wouldn't make sense for a distributed enterprise environment.

Ha ha ha.

>A simple shared file wouldn't make sense for a distributed enterprise environment.

Really? Here in enterprise land we have a technology called DFS Replication. It's actually only one of many such file replication technologies. It is hardly faultless, but I am quite certain, given your complete and utter noobishness, that it's world's beyond what you'll ever need.

Say enterprise a few more times though. It makes you really believable.

Cassandra it is.

This might also work.
http://commons.apache.org/configuration/apidocs/org/apache/commons/configuration/DatabaseConfiguration.html

Are we witnessing how "enterprise" actually gets made...

I made a central web-application with web-services for configuration data.

The web-application uses a SQL database to store all the values for different servers (parallel application servers) and different stages (test, acceptance, production).

Our install packages call the web-service to query their settings and put them into the configuration files, and register the installation of their version.

So the web-application provides per application a summary of all settings on all servers and which version is installed where. It also stores the installation packages. The build server uploads new versions to the central web-application, the installation software takes the package and installs it, the webservice provides the settings and the registration of installation.

Maybe I should make a commercial version :-)

Sadly, yes. Bot is a Java guy, so it makes sense.

"I made a central web-application with web-services for configuration data. "

We considered that also.  JSON for reads and writes.

It is possible to even have a simple REST based application sitting behind a light django application.

If you really want to go simple-brute-force, you can use a Perl program.

Worst case, you hard-code in a perl array the source-path and destination path, then have Perl 'scp' the source <machine>:<path> to the destination <machine>:<path>.

If there's not too many machines, you can have the operator enter the passwords as 'scp' prompts for them.  Or you can hard-code the username/passwords in the program.  Or you can hard-code the encrypted username/passwords in the program and decrypt them on the fly.

Next level up -- maintain a simple config file of source / dests, and have perl read THAT and execute 'scp' for you.

This IS one of the things Perl was created to simplify for people.

a config file?

If one of the params happens to be stuff to connect to the db ... that's a real good reason not to put it in a db ... ;-)

> If one of the params happens to be stuff to connect to the db ... that's a real good reason not to put it in a db ... ;-)

It was one of the reasons I used web-services instead of direct DB access: much easier to provide an URL that is valid over different domains (Test, Production etc.) than a DB connection.

Every server gets a registry setting with the url of the configuration server.

For secret settings (passwords) we used authentication methods over HTTPS, or certificate signing, be today we don't use passwords, restricted access is always governed by domain authentication of the accounts running the applications, so we no longer have sensitive configuration data.

I would use an SQL DB , if you already have those in your env.
I would use a file, if you already have distributed filesystems in your env.

I really like redis for this, but....I don't see introducing redis just to store these config key values.  People are going to reasonably ask, why all the overhead for a config file.

Unless, you want to get redis in the env to use it for something else in the future.  Then it is a cheap, safe was to get a toe in the redis waters.

ps

the fact and way you are asking makes you almost too stupid to live.  you really should quit that shitty 'enterprise' job you have as soon as practical.  It's rotting your brain.

"  People are going to reasonably ask, why all the overhead for a config file. "

It is not ONE config file.  It is hundreds or possibly thousands of configuration settings set in place for the last 20 years.

These settings are distributed across many servers.  Some in files, some in database.

In the J2EE world, architectures will use JNDI.  The issue with JNDI is that only the application server administrators can access the JNDI settings.

Database is an obvious choice but I was curious if something has trumped the database approach in the last couple of years.

Web.config can be used for read-only configuration values.  If you save values back to web.config you will cause IIS to restart your app.  This will cause problems with any active requests.

C# has a really nice Settings.settings interface.  I think it stores the values into an XML file somewhere. It works great.

Also, there is nothing wrong with storing the values in a relational database table.

YAML on disk and a hash map / dictionary in memory?

Both are super simple and effective.

I won't bring up XML.  I've been forced to use it for the disk store.  Not a big issue for small bits of data, but for large data sets it bloats incredibly.

We use a publish-subscribe messaging system, key-value storage. Both key and value are strings, but you can stuff anything there (using for instance XML or JSON encoding).

Bot, as you seem to be a Java guy, you could take a look at http://en.wikipedia.org/wiki/Java_Message_Service

Then either use some existing implementation, like Apache ActiveMQ, or implement your own solution.

>Also, there is nothing wrong with storing the values in a relational database table.

As Wayne has described, if you are using a database then sure -- presuming that you're fine with those settings being unavailable when the database is down, which is usually fine, then there's no problem.

In this case that isn't what Bot's saying. He is saying "What sort of fancy additional complexity and breaking points can I add to my stack...so I can distribute session timeout settings". Come on. A distributed key-value store to host very seldom-changed, read-infrequently settings? That sounds like the choices of someone dangerous enough to make an Enterprise System.

>a config file?
>
>If one of the params happens to be stuff to connect to the
>db ... that's a real good reason not to put it in a db ...
>;-)

This.

Also, if you want to share your database between test, dev, staging and live environments but change your settings, you shouldn't put your settings in a db.

Also, if you want some or all of your settings files managed in source control you shouldn't put it in the DB.

INI and YAML have the advantage of being human readable, but the disadvantage that they are still valid if a file gets chopped in half for some reason (this has happened to me before with weird and fucked up consequences). Also, no hierarchical info.

JSON has the advantage of being good for hierarchical data but not nice for humans to edit.

XML has the advantage of being ok for hierarchical data and ok for humans to edit (unless you created a really fucked up schema), but it has the disadvantage of being XML. Still, probably one of the only situations left in which I would say XML isn't such a bad idea.

>Cassandra it is.
>
>This might also work.
>http://commons.apache.org/configuration/apidocs/org/apache/commons/configuration/DatabaseConfiguration.html
>Bot Berlin
>February 27th, 2012 7:27pm


>Are we witnessing how "enterprise" actually gets made...


hahahahaha :)

I guess you missed the org.apache.commons.configuration.DatabaseConfiguration

Cassandra was a deflection.

Don't you mean org.apache.commons.configuration.DatabaseConfiguration.ReflectionAgent.FactoryBuilder.Builder.Builder.ModelAppCreatorBuilder.FactoryModel.EnterpriseArchitectsAreSoCool.DataConfurationManagerFactory.SettingsBuilder ?

Goddamn it wayne, your forum ruined my joke.

Bwahahaha. Had to inspect element to enjoy the joke, but worth it.

Enterprise Software is very very serious.

POST requests to whiteboardcam.txt2doc.nini.keyserver :)

I like webservice returning XML for this too.

It will scale well and should stand the test of time too.

NoSQL may be rightfully dead in five years.