Working with the Google Datastore emulator

 I do a good chunk of my business in Google App Engine; you package up your web application, send it to GCP, and then it takes care of scaling and uptime and all that stuff.

When I started out in 2014, I created my main application in Java because that was the least-crappy language that was supported.  However, in 2020, there are a whole lot more languages (in particular: Go).  I've slowly been working on porting my application from Java 8 to Go 1.14.  Along the way, I've run into some really annoying issues.

For today, I'm going to be focusing on the Datastore emulator.  In "old" App Engine (Java 8, Go 1.11, Python 2, etc.), they gave you a whole emulator suite.  Your application ran inside of that suite, and you had fake Google-based App Engine authentication, inbound e-mail, and a Datastore emulator that also had a web UI that you could use to see your entities and manipulate them.  The Datastore emulator's web UI wasn't as good as the current one that you get in production, but it was good enough to use for development.

Well, in "new" App Engine, the emulator suite is gone, and now you have to emulate or mock every aspect of App Engine that you plan on using.  It's not a huge deal, but it is a bit inconvenient.  In particular, you now have to start your own Datastore emulator.

It's easy to start:

gcloud config set project <your-project-id>;
gcloud beta emulators datastore start;

There are some environment variables that you'll need to export for the various libraries to detect and use instead of the production instance; run this to see them:

gcloud beta emulators datastore env-init;

That part is fine.

There are also two halfway-decent third party web UIs for the Datastore emulator:

1. https://github.com/GabiAxel/google-cloud-gui
2. https://github.com/streamrail/dsui

I fought for hours trying to figure out why either of those two web UIs didn't work.  Neither would show any namespaces (and thus, neither would show any entities).

The short answer is that despite what the Datastore emulator claims it's using for the project ID, the only thing that it actually uses is "dummy-emulator-datastore-project".

I got a hint about it by poking around in the emulator's data file, and I got some confirmation in this file, which is the only thing on the Internet at the time of this writing that references that string: https://code.googlesource.com/gocloud/+/master/datastore/datastore.go

So, if you start the Datastore emulator according to the instructions and either of those two web UIs aren't working, try setting the project ID to "dummy-emulator-datastore-project".

1. In "google-cloud-gui", you set the project ID in the UI when you hit the "+" button to create a new project.
2. In "dsui", you set the project ID using the "--projectId" flag.

Google App Engine cron jobs run as no user

Google App Engine (GAE) is a great platform to use when developing just about any new application.  I use it for a couple personal projects, as well as for some fire department apps (dispatch and inventory management).

Today I'd like to talk about two things: users and cron jobs.

Users

One of the really convenient things about GAE is that it has built-in support for Google authentication (no surprise there).  This means that you can let GAE take care of your sign-in system (and trust me, handling the single-sign-on 3-way handshake isn't all that fun).  With GAE, you easily go one of two routes:

1. Certain paths in your web application are automatically required to have a logged-in user.  If someone tries to access a path without being logged in, GAE will redirect him to a sign-in window and then bring him back when he's done.
2. Server-side, you can ask GAE for log-in and log-out URLs, and you can direct a user to these at any time to have him log in or out.

I personally prefer the second route because there's no strange redirection and all of my endpoints behave as I expect them to.  For example, if I have a JSON REST API, a user (who has not signed in yet) can access the endpoint and be given a normal 400-level error (in JSON) instead of being redirected to a Google sign-in page.  My REST clients much prefer this.

To see if a user is signed in, a Java Servlet can call the "getCurrentUser()" method of the "UserService" instance.  If the result is null, then no one is logged in.  Otherwise, you get a couple of helpful methods to tell you about the user:

1. "getEmail()"; this returns the user's e-mail address.
2. "getUserId()"; this returns a unique numeric ID for the user.
3. "isUserAdmin()"; this returns whether or not the user is an administrator of the GAE application.

For my apps, user authentication is whitelist style.  I check the "@" portion of the e-mail address to see if the user is in one of the domains that I care about (I typically build apps internal to an organization that uses Google's mail system), and I check the administrator status to grant administrator powers to my admins.

If someone tries to access a sensitive API endpoint without being logged in appropriately, I'll send back a 400-level error stating that the user is not signed in with an appropriate account.

Pretty easy stuff.

UserService userService = UserServiceFactory.getUserService();

User user = userService.getCurrentUser();

if( user == null ) {

   // There is no user logged in.

} else {

   // The user is logged in.

   System.out.println( "User is logged in: " + userService.isUserLoggedIn() );

   System.out.println( "User is administrator: " + userService.isUserAdmin() );

   System.out.println( "User:" );

   System.out.println( "   Auth Domain: " + user.getAuthDomain() );

   System.out.println( "   E-mail: " + user.getEmail() );

   System.out.println( "   Federated ID: " + user.getFederatedIdentity() );

   System.out.println( "   Nickname: " + user.getNickname() );

   System.out.println( "   User ID: " + user.getUserId() );
}

Warning: you cannot call "userService.isUserAdmin()" if the user is not already logged in.  If you try to, then it will throw an exception.

Cron Jobs

Another thing that GAE can do is schedule cron jobs.  Basically, these are page requests that are scheduled like normal Linux "cron" jobs.  So if you need to have some task performed regularly, create an endpoint for it and schedule a job to access that endpoint.

Cron jobs act as if an administrator is making the request, so they can access all paths with "admin" requirements.  Howerver, you cannot check this using "userService.isUserAdmin()" because cron jobs do not run as any particular user.

To determine if a request is coming from the cron scheduler, you have to check for the "X-Appengine-Cron" header.  This header cannot be faked (except by admins); if you try to set this header, GAE will quietly remove it by the time that it gets to your Servlet.

To detect a cron job, you have to check for the header and make sure that its value is "true".

String cronHeader = request.getHeader("X-Appengine-Cron");

if( cronHeader != null && cronHeader.compareTo("true") == 0 ) {

   log.info( "Cron service is making this request." );

}

Ultimately, if I'm checking to see whether a user is allowed to access a particular section, I go through these steps:

1. (Assume no access at all.)
2. No user is logged in.
1. Is the "X-Appengine-Cron" header set to "true"?  If so, then allow administrative access.
3. A user is logged in.
1. Is the user a GAE admin of the application?  If so, then allow administrative access.
2. Is the user's e-mail domain in the whitelist of basic user access?  If so, then allow basic access.

Google App Engine and Google Authentication

One of the things that I love about the year 2015 is "cloud computing"; in particular, I love that I can hand Google App Engine a Java project and it will host it, handle redundancy, auto-scale it, provide me a database, and do just about everything else that I could ever want.

However, security is still a major concern, and there are some applications that I work on where "leaking" some private data onto the public Internet is bad news.

User authentication

There are lots of ways to authenticate users at this point.  Years ago, we did everything ourselves (remember those days?).  Each application had its own user database with varying degrees of security, and passwords were being stolen and sold all the time.  Now we have things like OAuth, where we can pass off user authentication to other systems (which we have to trust), so we don't have to store anything more than an e-mail address.  If the OAuth server says that that e-mail address is legit and logged in, then it's legit and logged in.  This saves us time and money, since who wants to build and maintain a user authentication layer, anyway?

Google App Engine provides a pretty easy and awesome way to lock down your application to "signed in" users.  Here, "signed in" users can be one of two things:

1. Any user on Earth with a Google account; or
2. Any user in your Google Apps domain.

If you have a Google Apps domain, then with a couple of tweaks to both the domain and the app, then Google will make sure that only those people in the domain can log in to the app.  Otherwise, if you'll have to check for a particular domain from your logged-in users in your REST API calls (your app is RESTful, right?).  No biggie, either way.

(In this article, I'll be talking about the Java version of the Google App Engine SDK, so any files or calls will be related to that.)

To set up your app to force everything to require a logged-in user, you just need to update "web.xml" and add a "security-constraint" (obviously, you can play a lot with this):

<security-constraint>

<web-resource-collection>

<web-resource-name>site</web-resource-name>

<url-pattern>/*</url-pattern>

</web-resource-collection>

<auth-constraint>

<role-name>*</role-name>

</auth-constraint>

</security-constraint>

The "url-pattern" of "/*" means "MATCH ALL THE URLS!", and the "role-name" of "*" means "all users signed in with a Google account".

I can't log out!

I recently put together an application for a group where I don't own the Google Apps domain, so I couldn't set up the sweet domain-level restriction at the app level.  Like I said, it's no big deal, so at the top of my API class was a check for "@myspecialdomain.org" in the user's e-mail address (remember, Google already verified that it was a Google account, so I just needed to make sure that it was in the right domain).

I asked the group to try out the application, and half loved it; the other half couldn't see any data.  It turned out those people logged in with their personal accounts (not their organization-specific accounts), and now were stuck in an application where they had no access to any data.

What I needed was a "switch accounts" button (like all of the other Google applications), or at least a way to invalidate the user's session so that they could log in again (and use their organization account, this time).  Google App Engine's built-in authentication system does some magic, so the usual Google authentication guides don't apply.  I tried all of the stuff that people online were talking about (invalidating sessions, deleting cookies, etc.), but none of it worked, and it seemed like a hack anyway.

It turns out that Google App Engine provides a "UserService" class help deal with this kind of problem.  In particular, you may request a login URL and a logout URL.  Since my whole app requires the user to be logged in, all I needed to do was log the user out, and he would be immediately redirected to the Google login/pick-account screen again.

I added a simple API call to return some information about the current user (so I could show the "Logged in as ..." message), and that call returned an additional property for the logout URL (remember, this URL is generated by Google App Engine and isn't trivial).  When my UI loads, it makes an AJAX call for the user information, grabs the logout URL, and provides a "Logout" link for any users that need to log out.

Problem solved.

Here's how to get the logout URL:

UserService userService = UserServiceFactory.getUserService();

String logoutUrl = userService.createLogoutURL("/");

The "/" argument to "createLogoutUrl" is where the user should be redirected once he logs out.  In this case, I'm sending him right back to the main screen of the app, where he will be asked to log in again.

Additional resources

1. For more information on "web.xml" within Google App Engine, see this document.
2. To learn how to set up authentication with Google App Engine, see this article.