How about a NoSQL database for your front-end?!

In my working on various front-end systems over the years, one thing that I have continually found painful is trying to manage data. Shoving it in some random location, calling the server relentlessly, keeping state where I need not, storing it in DOM — gross. And as web applications continually get beefier and users expect more responsive, awesome interactivity, the front-end has had to take on more of the workload in working with data. Thanks to HTML5 we have the rather awesome localStorage, but I, personally, find the API to not be really ultra sweet, and what they do give us seems a little under powered, IMHO. I also find that this is not an uncommon issue and in managing data on the front-end numerous developers end up creating a bunch of boiler plate and state management all over their code and it gets messy, painful to look at, and even more painful to manage.

All of that said, here is a pretty rad solution: ku4collections! What is a ku4collection you might ask. Well, a ku4collection is a repository of documents, much like a NoSQL database on the back-end, but it is on the front-end. And, really, that’s it!

Why do you care? Well, because it makes working with data pretty damn easy. For example, say you are working with an admin portal that contains a table of people and each person record contains a person and their personal info. For the sake of this article let’s say that there are 4 people.

Name: John Richards
Title: Owner
Department: Executive
Salary: $30,000.00
Age: 30

Name: Erin Samualson
Title: Developer
Department: IT
Salary: $80,000.00
Age: 24

Name: Jennifer McMullen
Title: Project Manager
Department: IT
Salary: $60,000.00
Age: 25

Name: John Johnson
Title: Janitor
Department: Maintenance
Salary: $50,000.00
Age: 64

With ku4collections you could get this data from the server as JSON and stick it into a ku4collection for later use. So we will simply create a collection with name “persons,” initialize it with some data, and then save it to storage. (You can do this in a console window if you like. I will use Chrome. You can also follow along by grabbing the latest ku4js-data from github)

var persons = $.ku4collection("persons").init([
 {
   firstName: "John",
   lastName: "Richards",
   title: "Owner",
   department: "Executive",
   salary: 30000,
   age: 30
 }, 
 {
   firstName: "Erin",
   lastName: "Samualson", 
   title: "Developer",
   department: "IT",
   salary: 80000,
   age: 24
 },
 {
   firstName: "Jennifer",
   lastName: "McMullen", 
   title: "Project Manager",
   department: "IT",
   salary: 60000,
   age: 25
 },
 {
   firstName: "John",
   lastName: "Johnson", 
   title: "Janitor",
   department: "Maintenance",
   salary: 50000,
   age: 64
 }
]).save();

Now that we have a collection, let’s run some queries on it. With ku4collections you can run all kinds of expected operations including: find, insert, update, and remove.

So let’s say we wanted to find all of the entities in our collection. We could simply write:

persons.find()

and we would get back an array of all of our entities.

Image

Now you are probably saying, “Alright, that’s sort of cool.” Well, word of warning, this is about to get a lot cooler very fast!

Let’s say that I wanted to find all of the persons with name John. Well then I would find by a query

persons.find({"firstName": "John"});

This code is saying find all of the entities in the persons collection that have a firstName value of John. And what do you know? We get back both of the Johns!

Image

Now I hope that you are saying, “Okay, that’s cooler. Now I’m listening.”

Good, because this is where it starts getting awesome!

What if I needed both of the Johns but I wanted the oldest ones first? Well, then you would add an $orderby.

persons.find({"firstName": "John", "$orderby": { "age": -1 }});

If you use $orderby, you simply use -1 to order by descending and 1 to order by ascending.

Image

Pretty cool, huh? But wait, there’s more! (<– Gratuitous infomercial humor :{)})

Maybe you want to find all of the persons who are either Executives or in IT. Then you would use an $in.

persons.find({$in:{"department": ["Executive", "IT" ]}, "$orderby": { "age": -1 }});

Now with this query you get 3 results all ordered by descending age! Wicked cool!!!

Image

Well now what if we wanted to add more people? Simply insert them!

persons
 .insert({
   firstName: "Alex",
   lastName: "Garrison",
   title: "CIO",
   department: "Executive",
   salary: 330000,
   age: 42
 })
 .insert({
   firstName: "Alexa",
   lastName: "Rogers",
   title: "COO",
   department: "Executive",
   salary: 430000,
   age: 38
 }).save();

DON’T FORGET TO SAVE!!!

Now let’s run our last find query again

persons.find({$in:{"department": ["Executive", "IT" ]}, "$orderby": { "age": -1 }});

Now we have 5 results still ordered correctly!

Image

Let’s say this is actually not good, though. We shouldn’t have the Executives in this table. No biggie, we can just remove them.

persons.remove({"department": "Executive"}).save();

This query is saying: remove all of the entities that are in the Executive department

DON’T FORGET TO SAVE!

Now let’s run our last find query again

persons.find({$in:{"department": ["Executive", "IT" ]}, "$orderby": { "age": -1 }});

Now we only get 2 results, still ordered, of course, but the Executives are gone.

Image

I did notice in those results though that Erin’s last name is spelled incorrectly. Let’s fix it.

persons.update({"firstName": "Erin", "lastName": "Samualson"}, {"lastName": "Samuelson"}).save();

This query is saying: update all entities where the firstName is Erin and the lastName is Samualson and update the lastName to be Samuelson — with an “e”.

DON’T FORGET TO SAVE!

And then lets run our find query

persons.find({$in:{"department": ["Executive", "IT" ]}, "$orderby": { "age": -1 }});

And here we see expected results, though, now Erin’s last name is correct.

Image

I hope that by now you are saying to yourself, “Dude, this is pretty sweet!” And thank you if you are. There is one thing that I saved for last though, and IMHO it’s the best — JOINS! Yup, you can join collections too! Check this out.

Let’s create a departments table for our contrived example here:

var departments = $.ku4collection("departments").init([
 {
   "name": "IT",
   "floor": 5
 },
 {
   "name": "Maintenance",
   "floor": 1,
 },
 {
   "name": "Executive",
   "floor": 22
 }
]).save();

DON’T FORGET TO SAVE!

Okay, now lets also add those Executives back.

persons
 .insert({
   firstName: "John",
   lastName: "Richards",
   title: "Owner",
   department: "Executive",
   salary: 30000,
   age: 30
 })
 .insert({
   firstName: "Alex",
   lastName: "Garrison",
   title: "CIO",
   department: "Executive",
   salary: 330000,
   age: 42
 })
 .insert({
   firstName: "Alexa",
   lastName: "Rogers",
   title: "COO",
   department: "Executive",
   salary: 430000,
   age: 38
 }).save();

DON’T FORGET TO SAVE!

And now lets try joining our persons and departments tables.

var personDepartments = persons.join(departments, "department", "name");

Okay, now a lot has just happened with the statement above. The query is saying, join the persons collection and the departments collection on persons.department is equal to departments.name. When we joined our collections it is analogous to us joining SQL tables using aliases. That is, to get the data that we are looking for we will have to be mindful of the collection name and the data. So let’s say I want to find out what floor John Richards works on. I would do the following:

First, I would query for John in my joined collection.

var results = personDepartments.find({"persons.firstName": "John", "persons.lastName": "Richards"});

Now, this returned our results, but of course they are an array. So, lets get the one and only item in our results and then find the floor.

results[0]["departments.floor"];

And we will see that John Richards works on floor 22!

Image

You could do this all inline of course.

persons
 .join(departments, "department", "name")
 .find({"persons.firstName": "John", "persons.lastName": "Richards"})[0]
 ["departments.floor"];

Image

Alright, last thing to do is clean up our toys.

Enter $.ku4store(). Remember how we kept saving? Well this is some syntactic sugar that ku4js-data offers. What it is actually doing is saving to the $.ku4store(). There are all kinds of things to do with the $.ku4store(), but we’ll save that for another article. Before we clean up, though, let’s take a look at one.

During this article we had to save our collections to a var. This can actually be rather painful in a real webapp depending upon how you architect your solution. But what is really great is that you can get your collection by asking $.ku4store() for it. So, let’s try with one of your past queries. Let’s just find all people named John again and order them by descending age.

$.ku4store().read("persons").find({"firstName": "John", "$orderby": { "age": -1 }});

Image

And there you have it. Now you can access your collections anywhere. It gets even simpler and cooler in ku4js-webApp. That article to come soon!

Okay, then, let’s clean up our collections to free up some memory.

$.ku4store().__delete();

And there you go, ku4collections! I hope you find them useful, and just in case you didn’t grab the project yet, you can find it at ku4js-data on github.

Please, comment on and share this post with others if you enjoy it; follow @kodmunki on Twitter for regular updates on new developments, deployments, articles, and tools; and check out kodmunki™ on github for cool, managed, and tested kodmunki™ projects.

Thank you for reading, and as always, happy coding :{)}

12 thoughts on “How about a NoSQL database for your front-end?!

  1. Very cool! I’ll have to explore this a bit further. I think there are ways I can enhance user experience with an approach like this. I work with Oracle Application Express (APEX), which is a RAD tool for building web applications. All my data lives in an Oracle DB, however, the front-end is still very much HTML, CSS and Javascript.
    I think I can let the user massage and work with their data using ku4collections without roundtrips to the DB and only send data when ready. Even when data persistency is critical, a library like this should make life easier too.
    I’m looking forward to playing with it.

    1. Thanks for reading rimblas, and thank you for the comment! Those sound like perfect use cases. It would be excellent if you checked-in and shared your results!

      Best of luck to you in development! :{)}

      1. So, next follow-up question, if I may. What benefit do I get form ku4collections vs lodash?

      2. Thank you for your inquiry, rimblas!

        It is difficult to say what benefit you, specifically, get from ku4collections (ku4jQuery-data) “vs” lodash. What I can tell you is what has proven beneficial for us in using ku4collections, and more generally ku4*.

        For us, the benefit has been the ease in building maintainable, testable, domain-driven, OO, web-app front-ends that speak the language of our domain.

        What we have found beneficial with ku4collections, specifically, is that they lend themselves very nicely to data-driven aspects of these types of applications. Also, ku4collections enable us to leverage common design patterns, such as: specification patterns to query our collections, allowing for eloquent and verbose use and reuse of common domain policies.

        Furthermore, this has given rise to vastly leveraging IoC that exposes otherwise hidden members allowing them to be easily Unit Tested.

        Lastly, ku4collections are seamlessly integrated into ku4webApps framework. A framework that has proven to further enable us to develop quality applications, quickly, and easily while exposing the power of ku4collections.

        I hope that this answers your question, and that you find a library for your current project that will maximize your ROI and prove a solid, value-adding tool! If it happens to be ku4*, it would be awesome to see what you do with it!

        Best of luck!

        Thank you for reading, and happy coding! :{)}

  2. What exactly is jQuery being used for here? Are you loading the data into the DOM somewhere and then using jQuery to query against it as if it were part of the web page? It just seems odd to me that jQuery would be needed for something that’s data driven.

    1. Great observation Michael Oryl! And you are correct, there is no need for jQuery here. In fact, it plays no role in ku4Collection whatsoever!

      ku4collections is a part of a larger library: ku4jQuery-data, which is a project that is very much alive. ku4jQuery-data arose out of an earlier project called ku4j, which was highly dependent upon jQuery, but certainly lacked any of the power that ku4jQuery-data has today.

      Furthermore, ku4jQuery-data is a part of a larger suite of libraries that are known as ku4*. The ku4* libraries have been ported to Node and to Objective-C (yet to be released) and are called ku4node and ku4objc respectively. The jQuery port initially depended on jQuery for a few carry over items from ku4j. This meant that the project was compiled with jQuery to ensure the dependent components would still work.

      All of that said, “jQuery” is merely an artifact at this point, but very much a low-level dependency that still remains in the project, at its core. The ku4jQuery projects still leverage the global jQuery object, so without the existence of this object ku4jQuery-* will break. I am happy to say that this dependency is very close to being removed, at which time ku4jQuery-* will be renamed to express the removal of this current dependency, but we didn’t want something as cool as ku4collections to be kept from the masses due to this relatively insignificant, albeit inconvenient, truth.

      Thank you for your comment, and bringing to light a question that I am sure others had on their mind!

      I hope that you still find ku4collections a valuable tool to add to your belt.

      Best of luck in your current endeavors! :{)}

    1. Thank you for your comment, Matt Payne. Removing jQuery is certainly on the “To Do” list. Please, keep an eye out for updates to these libraries and be certain to stop back to kodmunki.wordpress.com. There is certain to be a post the moment the jQuery-less compilation is released.

      Happy coding! :{)}

  3. The Web app/programmer world desperately needs a good NoSql style database in the Browser. In my ideal world it’s api and capabilities would be similar to MongoDB, which is what I use on the Server. It is good to see you queries look Mongo’ish.

    localstorage is quite limiting in it’s maximum storage size. Are you considering pluggable backends so for example IndexedDB could be used.

    I look forward to seeing where you go with this project.

    1. Thank you for taking the time to comment, Neville Franks!

      It’s great to hear that there are others out there who yearn for a little more power in the realm of browser data!

      In answer to your question, “Are you considering pluggable backends…,” I am happy to say, most certainly! As you point out, there are certainly potential problems related to max storage sizes, and there is kodmunki™ development underway to leverage other storage means. What is also great, is that ku4* is very much open to extension, and so is ku4collections!

      If you are anxious to get started with IndexedDB storage, you could create a class that leverages IndexedDB and implements the ku4store interface. Then, simply create your ku4collection and pass your new store to the store() method like so:

      $.ku4collection().store(yourStoreHere);

      Now, you are using your IndexedDB store! That said, if you would rather wait that is cool, too! Just check back here for new posts that are certain to announce new available storage mechanisms.

      I hope that this information is valuable, and that if you do decide to implement an IndexedDB store, you stop back and tell us how it goes.

      Thank you for reading, and happy coding :{)}

Leave a reply to kodmunki™ Cancel reply