Tips and tricks from personal experience, followed by a list of study materials
Hi! You must be interested in passing the Google Cloud Professional Data Engineer Exam. Google recommends that you have 3+ years of experience before attempting the exam. However, I think that if you have some experience with other cloud providers, databases and SQL, you can still do it, namely because GCP is much more intuitive than its competitors (in my humble opinion).
Unlike other certifications, there isn’t a regimented coursebook or training manual. That is because Google expects you to be a practitioner and know most things…
Anomalies and outliers and how to find them.
When I started to write this article the first thing I did was to look for formal definitions of anomaly and outlier. Turns out that there isn’t a consensus on the matter. Every field would have (annoying) a slightly different opinion:
How to get rid of Jupyter Notebook and use Jupyter Lab instead.
We at DataSparQ love Kubeflow. But we don’t like the default Notebook options that are on the menu:
“You can have any colour you want, as long as it’s black”
They all come with:
apt getis the only way to install packages like
How to return meaningful responses when something goes wrong.
Exceptionsin your Flask app
Has something like this ever happened to you?
3 simple steps to build a simple CI/CD using your existing Kubernetes cluster
A diagram and the source code are at the bottom of the page.
Houston is our home grown server-less solution for workflow management. It has a couple of…
… when you want Helm without Helm
We wanted parameterised Kubernetes deployments but Helm was too complicated to integrate with our CI/CD. So we solved the problem with Jinja Templates and a python script running in Cloud Build.
Code is available as a Github gist: here
We have a product called Houston that simplifies workflows. It has an API, a web interface, and a couple of other components that all live in Kubernetes. We wanted to have a continuous integration/deployment where testing, staging and production were all running on the same cluster but in different namespaces. …
If you ever looked at an enterprise network traffic, you quickly find out that the majority of the traffic is to a handful of domains like Google, Apple, Facebook, Instagram and of course Netflix (makes you wander how much work exactly gets done there).
And if your job is to find anomalous traffic/connections, it is very useful to get rid of those big an noisy domains by having some kind of whitelist. I am not going to insult the readers intelligence by listing unrealistic and broken solutions. The way I went about it is to use domain rankings. I have…
I recently decided to give Jupyter Lab a try as an alternative to RStudio. And there were a couple of basic requirements I had:
What seemed a trivial task quickly turned in more than a day of frustration. I managed to get the Jupyter container up and running behind NGINX, but I could not get any code running in the notebooks — I could not connect to the Python kernel.
After several hours of trying chasing the wrong questions…
This is part 2 of a series of articles that explore the prospective salaries and living expenses across several countries and cities. If you want to read part 1:
In the previous part, we talked about salaries in the driven industry for California, UK and Germany. I also touched on the taxes of those locations and came up with a net annual salary estimate. Next I would like to talk about expenses in those places.
This is where is gets really messy and complicated. I have given up on the idea of achieving high validity, but that doesn’t have to…
Couple of moths ago I went on a journey to find myself one of those elusive creatures called jobs. For those of you who have been in my shoes (or still are), you are painfully aware how time consuming this process can be. It can often feel overwhelming and futile, and if you lack the emotional capacity, it can get the best of you.
For reasons I will not dwell upon in this article, I have decided that I will take the challenge up a notch or two - I decided to apply to multiple countries simultaneously without ever having…
When the machines take over, I will be on the winning side 🤖