Nobody likes to be called a dummy by a dummy.

Database but not a database, updatable

What kind of system do you use (or smart people) use for a database of records that are used for distributed configuration settings?

A key value store would work but I hear those are normally used for caching and not as an updatable system for configuration settings.

I asked this question on StackOverflow and the overwhelming response was to use a SQL database.

SQL database is also fine but that isn't the role for a SQL database.

...
Imagine:

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200
Permalink Bot Berlin 
February 27th, 2012 4:23pm
.ini
Permalink bpd 
February 27th, 2012 4:33pm
> SQL database is also fine but that isn't the role for a SQL database.

Isn't it?  What makes it unsuitable?  You haven't really provided enough information about what you're going to store, how many records we're talking about, or even why it's distributed.

If your configuration is well structured, wouldn't that map exactly to a database table (columns as settings, values as selections).
Permalink Send private email Wayne 
February 27th, 2012 4:36pm
Why not a relational DB?
Permalink Shylock, not Denman 
February 27th, 2012 4:37pm
Distributed config info?

Another vote for INI file.

(as long as there's no hierarchical information stored)
Permalink Send private email xampl9 
February 27th, 2012 4:40pm
In a web environment.
Permalink Bot Berlin 
February 27th, 2012 4:43pm
Still not enough information.  Is there some reason why you wouldn't want to go SQL DB?  Does this project not use a DB at all?
Permalink Send private email Wayne 
February 27th, 2012 4:45pm
"Is there some reason why you wouldn't want to go SQL DB? "

It does.

A SQL DB is normally used for relational application data.

Is there something web/distributed exclusively for updating configuration settings.

With a DB, I have to go through the work of connecting to the database, querying my data.

I was hoping in 2012 there was an online system used for updating configuration settings.  Something that is fast, persistent and allows for periodic updates.

...

Ironically, distributed key stores like MongoDB and Cassandra are used for caching solutions not really for config settings.
Permalink Bot Berlin 
February 27th, 2012 4:53pm
"Is there some reason why you wouldn't want to go SQL DB? "

Nothing really, I was just curious if something else existed.
Permalink Bot Berlin 
February 27th, 2012 4:54pm
Why not Zoidberg?
Permalink df 
February 27th, 2012 5:01pm
MongoDB is a document store. Both Cassandra and MongoDB are persistent and are seldom used for caching. Caching generally uses things like Redis or Memcache/Membase.
Permalink df 
February 27th, 2012 5:03pm
It sounds like you're searching for a solution to a problem you also haven't discovered. For your basic description as a simple shared file sounds overwhelming fine.
Permalink df 
February 27th, 2012 5:04pm
> A SQL DB is normally used for relational application data.

That might be over thinking it.  If you can structure it into a table, then a SQL DB is a reasonable choice.  The data doesn't really have to be "relational".  There's lot of data that doesn't fit well into a table -- at that point a SQL DB might be a poor choice.

> Is there something web/distributed exclusively for updating configuration settings.

Seems a bit specific.  You could use any of the NoSQL solutions that include persistence. 

> With a DB, I have to go through the work of connecting to the database, querying my data.

Isn't that true of any client/server solution.  If I'm using, for example, memcache as a caching layer I still need to connect to it and query the data.

Seems you want something too specific to actually exist.
Permalink Send private email Wayne 
February 27th, 2012 5:06pm
This is a simple property file.  Normally in the Java world, you may include these settings are part of your web app deployment.

So you deploy your property settings as:


MyWebAppOnTheFileSystemAsReadOnly/MySettings.properties:

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200

This is fine but those settings are read-only and can only get updated when you redeploy your web application.

...

So now you move the settings to a random location on the filesystem:


/usr/local/var/MySettings.properties

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200

...
This approach makes it easier to update the settings but you have set it up so only an admin will have to edit the file manually using vi or some unix editor.

The application developer or administrator can't update the configuration settings online.

...

So then you move your application to database

MyConfigTable@MYDB
user=confmaster
pwd=configuration

select * from MyConfigTable

TIMEOUT_SETTING=100
TIMEOUT_SETTING2=200


...

That approach also works but you are using a relational database for simple key value system.

Does anyone here actually work in enterprise development?
Permalink Bot Berlin 
February 27th, 2012 5:13pm
"Isn't that true of any client/server solution. "

True, but you avoid the time for authentication and the database lookup, etc.

NoSQL can be faster than a large database.
Permalink Bot Berlin 
February 27th, 2012 5:15pm
If the app uses DB already, then yes. If not, it's an overkill. +1 for ini file
Permalink sierra 
February 27th, 2012 5:15pm
Wayne,

What NoSQL solutions are people using for this type of task.

MongoDB, MemcacheDB, CouchDB, Cassandra?

And J2EE has Jndi.
Permalink Bot Berlin 
February 27th, 2012 5:16pm
"NoSQL can be faster than a large database."

Ugh.
Permalink df 
February 27th, 2012 5:16pm
Step back, everyone, Bot is doing "enterprise" work here.

Ha ha ha.
Permalink df 
February 27th, 2012 5:17pm
"For your basic description as a simple shared file "

It was a response to your comment.  A simple shared file wouldn't make sense for a distributed enterprise environment.
Permalink Bot Berlin 
February 27th, 2012 5:20pm
Ha ha ha.
Permalink Bot Berlin 
February 27th, 2012 5:20pm
>A simple shared file wouldn't make sense for a distributed enterprise environment.

Really? Here in enterprise land we have a technology called DFS Replication. It's actually only one of many such file replication technologies. It is hardly faultless, but I am quite certain, given your complete and utter noobishness, that it's world's beyond what you'll ever need.

Say enterprise a few more times though. It makes you really believable.
Permalink df 
February 27th, 2012 5:23pm
Cassandra it is.

This might also work.
http://commons.apache.org/configuration/apidocs/org/apache/commons/configuration/DatabaseConfiguration.html
Permalink Bot Berlin 
February 27th, 2012 5:27pm
Are we witnessing how "enterprise" actually gets made...
Permalink Send private email Wayne 
February 27th, 2012 5:27pm
I made a central web-application with web-services for configuration data.

The web-application uses a SQL database to store all the values for different servers (parallel application servers) and different stages (test, acceptance, production).

Our install packages call the web-service to query their settings and put them into the configuration files, and register the installation of their version.

So the web-application provides per application a summary of all settings on all servers and which version is installed where. It also stores the installation packages. The build server uploads new versions to the central web-application, the installation software takes the package and installs it, the webservice provides the settings and the registration of installation.

Maybe I should make a commercial version :-)
Permalink Attila 
February 27th, 2012 5:29pm
Sadly, yes. Bot is a Java guy, so it makes sense.
Permalink df 
February 27th, 2012 5:29pm
"I made a central web-application with web-services for configuration data. "

We considered that also.  JSON for reads and writes.

It is possible to even have a simple REST based application sitting behind a light django application.
Permalink Bot Berlin 
February 27th, 2012 5:32pm
If you really want to go simple-brute-force, you can use a Perl program.

Worst case, you hard-code in a perl array the source-path and destination path, then have Perl 'scp' the source <machine>:<path> to the destination <machine>:<path>.

If there's not too many machines, you can have the operator enter the passwords as 'scp' prompts for them.  Or you can hard-code the username/passwords in the program.  Or you can hard-code the encrypted username/passwords in the program and decrypt them on the fly.

Next level up -- maintain a simple config file of source / dests, and have perl read THAT and execute 'scp' for you.

This IS one of the things Perl was created to simplify for people.
Permalink SaveTheHubble 
February 27th, 2012 5:33pm
a config file?

If one of the params happens to be stuff to connect to the db ... that's a real good reason not to put it in a db ... ;-)
Permalink eek 
February 27th, 2012 5:39pm
> If one of the params happens to be stuff to connect to the db ... that's a real good reason not to put it in a db ... ;-)

It was one of the reasons I used web-services instead of direct DB access: much easier to provide an URL that is valid over different domains (Test, Production etc.) than a DB connection.

Every server gets a registry setting with the url of the configuration server.

For secret settings (passwords) we used authentication methods over HTTPS, or certificate signing, be today we don't use passwords, restricted access is always governed by domain authentication of the accounts running the applications, so we no longer have sensitive configuration data.
Permalink Attila 
February 27th, 2012 5:46pm
I would use an SQL DB , if you already have those in your env.
I would use a file, if you already have distributed filesystems in your env.

I really like redis for this, but....I don't see introducing redis just to store these config key values.  People are going to reasonably ask, why all the overhead for a config file.

Unless, you want to get redis in the env to use it for something else in the future.  Then it is a cheap, safe was to get a toe in the redis waters.
Permalink line noise 
February 27th, 2012 7:29pm
ps

the fact and way you are asking makes you almost too stupid to live.  you really should quit that shitty 'enterprise' job you have as soon as practical.  It's rotting your brain.
Permalink line noise 
February 27th, 2012 7:30pm
"  People are going to reasonably ask, why all the overhead for a config file. "

It is not ONE config file.  It is hundreds or possibly thousands of configuration settings set in place for the last 20 years.

These settings are distributed across many servers.  Some in files, some in database.
Permalink Bot Berlin 
February 27th, 2012 7:36pm
In the J2EE world, architectures will use JNDI.  The issue with JNDI is that only the application server administrators can access the JNDI settings.

Database is an obvious choice but I was curious if something has trumped the database approach in the last couple of years.
Permalink Bot Berlin 
February 27th, 2012 7:39pm
Web.config can be used for read-only configuration values.  If you save values back to web.config you will cause IIS to restart your app.  This will cause problems with any active requests.

C# has a really nice Settings.settings interface.  I think it stores the values into an XML file somewhere. It works great.

Also, there is nothing wrong with storing the values in a relational database table.
Permalink Fan boy 
February 27th, 2012 9:26pm
YAML on disk and a hash map / dictionary in memory?

Both are super simple and effective.

I won't bring up XML.  I've been forced to use it for the disk store.  Not a big issue for small bits of data, but for large data sets it bloats incredibly.
Permalink Just an idea... 
February 28th, 2012 12:11am
We use a publish-subscribe messaging system, key-value storage. Both key and value are strings, but you can stuff anything there (using for instance XML or JSON encoding).

Bot, as you seem to be a Java guy, you could take a look at http://en.wikipedia.org/wiki/Java_Message_Service

Then either use some existing implementation, like Apache ActiveMQ, or implement your own solution.
Permalink Io 
February 28th, 2012 3:11am
>Also, there is nothing wrong with storing the values in a relational database table.

As Wayne has described, if you are using a database then sure -- presuming that you're fine with those settings being unavailable when the database is down, which is usually fine, then there's no problem.

In this case that isn't what Bot's saying. He is saying "What sort of fancy additional complexity and breaking points can I add to my stack...so I can distribute session timeout settings". Come on. A distributed key-value store to host very seldom-changed, read-infrequently settings? That sounds like the choices of someone dangerous enough to make an Enterprise System.
Permalink df 
February 28th, 2012 9:02am
>a config file?
>
>If one of the params happens to be stuff to connect to the
>db ... that's a real good reason not to put it in a db ...
>;-)

This.

Also, if you want to share your database between test, dev, staging and live environments but change your settings, you shouldn't put your settings in a db.

Also, if you want some or all of your settings files managed in source control you shouldn't put it in the DB.

INI and YAML have the advantage of being human readable, but the disadvantage that they are still valid if a file gets chopped in half for some reason (this has happened to me before with weird and fucked up consequences). Also, no hierarchical info.

JSON has the advantage of being good for hierarchical data but not nice for humans to edit.

XML has the advantage of being ok for hierarchical data and ok for humans to edit (unless you created a really fucked up schema), but it has the disadvantage of being XML. Still, probably one of the only situations left in which I would say XML isn't such a bad idea.
Permalink Colm 
February 28th, 2012 9:32am
>Cassandra it is.
>
>This might also work.
>http://commons.apache.org/configuration/apidocs/org/apache/commons/configuration/DatabaseConfiguration.html
>Bot Berlin
>February 27th, 2012 7:27pm


>Are we witnessing how "enterprise" actually gets made...


hahahahaha :)
Permalink Colm 
February 28th, 2012 9:43am
I guess you missed the org.apache.commons.configuration.DatabaseConfiguration

Cassandra was a deflection.
Permalink Bot Berlin 
February 28th, 2012 9:47am
Don't you mean org.apache.commons.configuration.DatabaseConfiguration.ReflectionAgent.FactoryBuilder.Builder.Builder.ModelAppCreatorBuilder.FactoryModel.EnterpriseArchitectsAreSoCool.DataConfurationManagerFactory.SettingsBuilder ?
Permalink Colm 
February 28th, 2012 9:51am
Goddamn it wayne, your forum ruined my joke.
Permalink Colm 
February 28th, 2012 9:52am
Bwahahaha. Had to inspect element to enjoy the joke, but worth it.
Permalink df 
February 28th, 2012 10:26am
Enterprise Software is very very serious.
Permalink Bot Berlin 
February 28th, 2012 10:33am
POST requests to whiteboardcam.txt2doc.nini.keyserver :)
Permalink trollop 
February 28th, 2012 5:52pm
I like webservice returning XML for this too.

It will scale well and should stand the test of time too.

NoSQL may be rightfully dead in five years.
Permalink JoC 
March 1st, 2012 5:04pm
http://zookeeper.apache.org/
Permalink Nico 
March 6th, 2012 7:56am

This topic is archived. No further replies will be accepted.

Other topics: February, 2012 Other topics: February, 2012 Recent topics Recent topics