Developer Experience beyond the IDE with Ganymede

When you think about writing code, you probably picture VS Code, a dark mode terminal, and getting your code to "compile". While the authoring of code is a critical part of the process, the subsequent steps of unit testing, peer review, and deployment are just as important. With Ganymede's new support for committing code via API endpoints, your team can manage and deploy the code that runs on Ganymede in the same way as every other piece of code that you use. 

Here's how your team can use Ganymede to manage data pipelines without toiling through the setup of cloud and application infrastructure. The workflow below presumes you have walked through the steps for using your own repo on Ganymede.

Start by outlining your data pipeline goals.  Will it be triggered by an ELN event (like in Benchling), a regularly scheduled run (hourly, every 15 minutes), or a Ganymede Agent automatically collecting data files from your lab instrument? 

Decide where the output of the Flow goes.  Will your data be output into a dashboard built in Ganymede or another BI tool, written to a data historian, or sent via email to alert your team when new or out of compliance data is received? Your stakeholders likely have many ideas, and using Ganymede is the best way for you to build and iterate fast.

Create a Flow using Ganymede’s UI once you’ve identified the data pipeline you want to run. Each node in the Flow should represent a discrete processing step where it's useful to store the results in the tables of a data lake. Sync the Flow structure to your repo in a pull request to start coding.

Write your data pipeline logic and processing steps in your IDE. Want to reuse code? Add it to a ‘lib’ folder so it’s accessible to any node or Flow in your environment. Once your logic looks good, you can add unit tests to check for edge cases and confirm the correctness of your change.

Run automated checks against your code before your co-workers take a look at your change. These checks can include your regular unit tests, integration tests that use the Ganymede API to fetch representative files or table data, or running Flows in a dev environment with your proposed updates deployed.. Pre-review testing ensures your changes are clean and lets your reviewers focus on the structure, naming, and purpose of your code instead of searching for bugs.

Once your code is approved, commit your changes.  Your repo automation sends code to Ganymede, and it’s ready to run. No lengthy deployments or mysterious container build errors. Your lab data can flow freely, and now you can focus on building software for science and not to make some other software happy.

If you'd like to chat about any of the above, have ideas on how to improve it, or questions on how to automate your team's workflow, reach out to us and we'd be happy to help!