From aca340731aa719cd5720161f8af458bd741718ea Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Sun, 3 Nov 2024 10:34:02 +0000 Subject: [PATCH] remove all blog content --- ...2020-06-16-how-information-is-organized.md | 134 ------ articles/2020-06-24-three-questions-to-ask.md | 82 ---- articles/2020-07-02-allow-yourself-to-fail.md | 74 ---- .../2020-07-14-cron-is-better-than-a-timer.md | 54 --- articles/2020-12-09-upsert-in-dhis2.md | 56 --- .../2021-02-03-hosted-or-local-deployment.md | 112 ----- articles/2021-02-17-syncing-options.md | 97 ----- articles/2021-05-24-commcare-events.md | 383 ------------------ ...2021-07-05-wrapping-my-head-around-jobs.md | 300 -------------- ...g-multistage-docker-builds-using-buildx.md | 207 ---------- .../2021-10-15-webpack-to-esbuild-part1.md | 215 ---------- ...10-22-testing-react-app-with-jest-hound.md | 290 ------------- ...script-helped-me-better-understand-jobs.md | 254 ------------ articles/2022-06-07-workflow-automation.md | 170 -------- articles/2022-09-19-auth-security.md | 79 ---- blog/2020-05-31-the-new-openfn-blog.md | 26 -- ...lding-Integrated-Systems-That-Just-Work.md | 119 ------ blog/2020-06-04-Technology-Isnt-The-Answer.md | 89 ---- blog/2020-06-09-enabling-scale-at-myagro.md | 87 ---- blog/2020-06-10-iKapaData-Case-Study.md | 63 --- blog/2020-06-25-SwissTPH-case-study.md | 60 --- blog/2020-07-01-KGVK-case-study.md | 57 --- ...african-school-of-excellence-case-study.md | 70 ---- blog/2020-07-01-airport-case-study.md | 62 --- .../2020-07-08-caris-foundation-case-study.md | 87 ---- blog/2020-07-13-x-runner-case-study.md | 72 ---- blog/2020-07-26-sinapis-case-study.md | 71 ---- blog/2020-07-27-nalibali-case-study.md | 75 ---- ...-09-interoperability_for_case_referrals.md | 108 ----- blog/2021-03-30-digital-global-good.md | 186 --------- ...-30-processes-and-open-source-as-choice.md | 159 -------- ...he-case-for-health-information-exchange.md | 63 --- ...eguard-the-world\342\200\231s-wildlife.md" | 48 --- ...Monitoring-program-enrollment-in-Brazil.md | 116 ------ blog/2023-03-01-open-human.md | 128 ------ blog/2023-04-13-lightning-beta.md | 93 ----- blog/2023-06-01-open-function-group.md | 143 ------- blog/2023-08-14-OS4H-case-study.md | 86 ---- ...Monitoring-child-protection-in-Cambodia.md | 57 --- blog/2024-03-21-kenya-travelogue.md | 203 ---------- docusaurus.config.js | 44 -- src/pages/index.js | 2 +- 42 files changed, 1 insertion(+), 4880 deletions(-) delete mode 100644 articles/2020-06-16-how-information-is-organized.md delete mode 100644 articles/2020-06-24-three-questions-to-ask.md delete mode 100644 articles/2020-07-02-allow-yourself-to-fail.md delete mode 100644 articles/2020-07-14-cron-is-better-than-a-timer.md delete mode 100644 articles/2020-12-09-upsert-in-dhis2.md delete mode 100644 articles/2021-02-03-hosted-or-local-deployment.md delete mode 100644 articles/2021-02-17-syncing-options.md delete mode 100644 articles/2021-05-24-commcare-events.md delete mode 100644 articles/2021-07-05-wrapping-my-head-around-jobs.md delete mode 100644 articles/2021-10-08-improving-multistage-docker-builds-using-buildx.md delete mode 100644 articles/2021-10-15-webpack-to-esbuild-part1.md delete mode 100644 articles/2021-10-22-testing-react-app-with-jest-hound.md delete mode 100644 articles/2021-10-29-how-learning-javascript-helped-me-better-understand-jobs.md delete mode 100644 articles/2022-06-07-workflow-automation.md delete mode 100644 articles/2022-09-19-auth-security.md delete mode 100644 blog/2020-05-31-the-new-openfn-blog.md delete mode 100644 blog/2020-06-04-Building-Integrated-Systems-That-Just-Work.md delete mode 100644 blog/2020-06-04-Technology-Isnt-The-Answer.md delete mode 100644 blog/2020-06-09-enabling-scale-at-myagro.md delete mode 100644 blog/2020-06-10-iKapaData-Case-Study.md delete mode 100644 blog/2020-06-25-SwissTPH-case-study.md delete mode 100644 blog/2020-07-01-KGVK-case-study.md delete mode 100644 blog/2020-07-01-african-school-of-excellence-case-study.md delete mode 100644 blog/2020-07-01-airport-case-study.md delete mode 100644 blog/2020-07-08-caris-foundation-case-study.md delete mode 100644 blog/2020-07-13-x-runner-case-study.md delete mode 100644 blog/2020-07-26-sinapis-case-study.md delete mode 100644 blog/2020-07-27-nalibali-case-study.md delete mode 100644 blog/2021-02-09-interoperability_for_case_referrals.md delete mode 100644 blog/2021-03-30-digital-global-good.md delete mode 100644 blog/2021-07-30-processes-and-open-source-as-choice.md delete mode 100644 blog/2021-10-20-the-case-for-health-information-exchange.md delete mode 100644 "blog/2022-06-28-Data-Integration-to-better-safeguard-the-world\342\200\231s-wildlife.md" delete mode 100644 blog/2022-10-06-Monitoring-program-enrollment-in-Brazil.md delete mode 100644 blog/2023-03-01-open-human.md delete mode 100644 blog/2023-04-13-lightning-beta.md delete mode 100644 blog/2023-06-01-open-function-group.md delete mode 100644 blog/2023-08-14-OS4H-case-study.md delete mode 100644 blog/2023-11-07-Monitoring-child-protection-in-Cambodia.md delete mode 100644 blog/2024-03-21-kenya-travelogue.md diff --git a/articles/2020-06-16-how-information-is-organized.md b/articles/2020-06-16-how-information-is-organized.md deleted file mode 100644 index 42d3847b4945..000000000000 --- a/articles/2020-06-16-how-information-is-organized.md +++ /dev/null @@ -1,134 +0,0 @@ ---- -layout: post -title: 'How Information Is Organized... In Organizations' -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [how-to, tips] -image: /img/informationorganized.jpg -featured: false ---- - -#### Does your organization's information have an underlying structure? Try this exercise using boxes and crow's feet. - -This article was originally posted by Taylor Downs, Head of Product, on -[The OpenFn Founder's blog](https://medium.com/@taylordowns2000) as "The power -of crow's feet." - -It’s Saturday Morning in Cape Town and I’ve just spent an hour talking about how -a non-profit is organized. I thought I was getting into a technical -discussion—I’ve been doing system architecture discussions for years—but what we -ended up talking about was how this NGO thinks. - - - -This engagement is largely about mapping an already existing “people & paper” -based system to technology. Vera Solutions will build a system for this client -using Open Data Kit for field data collection and Salesforce.com for the -management “back-end”. Because we’re not explicitly being asked to help redesign -processes at this organization, the client is “telling us how things are”, then -expecting us to create a relational database model that facilitates -business-as-usual, only in a more efficient, digital way. Seems reasonable. - -This organization runs multiple programs focusing on a handful of strategic -objectives. They coordinate various activities in their target communities and -report on those activities against numerous (sometimes overlapping) indicators. -Sound familiar? As we get to the 3rd explanation of these programs, and the 11th -iteration of the system schema, it hits me… - -### Drawing a "relational object model" sounds technical, but it's actually an exercise in clear communication... and everyone can benefit from it. - -When we—human beings—wrestle with complex problems (like managing lots of -programs, other humans, community stakeholders, etc.) we have the capacity to -trick ourselves into thinking that we have wrapped our heads around a system -(for clinic registration, for after-school education, etc.) when, in fact, we’re -engaging in mental hand-waving and are simply ‘papering-over’ sections which are -secretly not just complicated, but totally incongruous with other parts of the -system. We can make ourselves believe that our logic is sound because we want it -to be sound, when in reality the organization might be held together by good -people, not good, clear, defined processes. By learning a couple of key -concepts, it’s possible for non-technical people to articulate their thoughts -clearly using “boxes” and “crow’s feet” and see whether or not there is an -underlying structure to their organization’s information. - -### By forcing yourself to reduce complex systems to sketches containing only two elements, you’ll be able to detect important conflicts and confusions in how you think about your organization that you might otherwise miss. - -If you can’t diagram the information structure in your organization using boxes -and crow’s feet, it’s a smell that something isn’t quite right (or at least that -something isn’t easily scalable… more on this later!). Let me show you the tools -in the toolbox and then wrap up by waxing poetic on people, processes, and -technology. - -### Boxes and crow's feet - -![]({{ site.baseurl }}/assets/images/box5.png) - -The box is my favorite. It represents an entity in your data system. Entities -(like `teachers`) have attributes (like `name`, `phone number`, `date of birth`, -`gender`, etc.) Some people like thinking of entities as simple forms. The -“Teacher Registration Form” will ask for the teacher’s name, phone number, -gender, etc. These are the fields on your teacher entity. By submitting one of -these forms, you’ll add a new teacher to your database. If you’re an Excel -person, the attributes are columns in your `teachers` table. - -![]({{ site.baseurl }}/assets/images/crowsfeet2.png) - -The crow’s foot is my second favorite. It’s used to show relationships between -entities. We know that teachers are related to the sessions that they conduct. -(And `session` might be another entity, with fields like `date`, -`subject taught`, and `venue`, to name just a few.) The crow’s foot allows us to -specify exactly how they are related. On that session entity, we’ll need to -specify the name (or ID) of the teacher who led it. On the teacher entity, -however, there won’t be a field to specify the name or ID of the session… -because a single teacher can lead MANY sessions. This is a one-to-many -relationship. The crow’s foot (that little three-pronged fella) denotes the -many. One teacher can have many sessions. One session, however, can only have -one teacher. See the diagram below. - -![]({{ site.baseurl }}/assets/images/objectmodel3.png) - -If we focus just on `teacher` and `session` and think back to MS Excel, we can -envision a `teachers` table and a `sessions` table. Let’s put them on different -sheets in the same workbook. On the `teachers` table, there is no column for -`session`, but on the `sessions` table, there _is_ a column for `teacher ID`. -We’ve just established a one-to-many relationship. - -Next time, we’ll talk about what’s going with the `attendance` entity above. -It’s sometimes called a “junction object” or a “join table”, and it’s what -allows MANY students to be related to MANY sessions. I’ll write more on this -next time, but there is no magic going on, no technicalities here. The way that -many students are related to many sessions is through this Real World Concept -that we call `attendance`. Attendance is what happens when a student shows up at -a session. It’s so important to get the language right in these discussions, and -make sure that you’re talking about real-world concepts. - -### Relational object models with lots of confusing terms are not "technical". They are "bad." - -Remember that as you start to put pen to paper. And allow yourselves time (and -multiple drafts) to get the boxes and terminology right. Understanding -relational object modelling is an incredibly powerful way to organize a company. -As I said before, if you can’t model it with boxes and arrows, it’s a smell that -something might not be conceptually sound. - -### A disclaimer and some thoughts on scaling: - -Some organizations do amazing work without good conceptual systems. They rely on -humans, instinct, improvisation, nouse, and other not-so-clearly-defined things. -They might do really great work. They might get the job done. But they need to -be well aware of their condition and face it head on. If you can’t systematize -your program implementation processes, then you need to focus tremendous effort -on finding and retaining the right people. - -A friend once told me that “people are not scalable.” I couldn’t agree more, and -defend my earlier stance that if your organization’s information structure can’t -be defined with boxes and crow’s feet, it may be very hard for you to scale -responsibly. However, if you can create a ruthlessly efficient, world-class -“people operations” system (recruitment, training, management, compensation, HR, -etc.) that ensures you’ve always got the right people to figure things out you -might be better off than those operating a well defined assembly-line with -interchangeable parts. Alas, the middle way is probably the best. - -That’s all for now. More soon. - -_Need help organizing or scaling your organization's information or process -flows? Contact our team of ICT4D specialists at ._ diff --git a/articles/2020-06-24-three-questions-to-ask.md b/articles/2020-06-24-three-questions-to-ask.md deleted file mode 100644 index 581a5633d472..000000000000 --- a/articles/2020-06-24-three-questions-to-ask.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -layout: post -title: 'To Automate or Not to Automate? Ask Yourself These 3 Questions.' -author: Aleksa Krolls -author_url: https://github.com/aleksa-krolls -author_image_url: https://avatars.githubusercontent.com/aleksa-krolls -tags: [tips] -featured: true ---- - -Automation can save time, unlock critical resources, and enable scale–but it -typically requires investment to set up. Wondering whether you should automate -your processes? Ask yourself these 3 questions. - - - -### Our partners use [OpenFn](https://openfn.org) automation solutions to drive efficiency and scale their processes, delivering integrated digital systems that work better, faster, and together. - -To date, we have worked with 43 social sector organizations that operate across -sectors–from health, education, and agriculture, to livelihoods and emergency -response. Over the last 6 years, OpenFn has been implemented worldwide for a -wide range of use cases, including building real-time data monitoring systems, -streamlining data cleaning pipelines, securely exchanging sensitive information, -and automating routine processes like uploading indicator results, sending -SMS/email alerts, making mobile payments, -[and more](https://openfn.org/solutions). - -By connecting any app, OpenFn can integrate and automate all apps within a -digital ecosystem. However, a question that we frequently ask our partners is: - -#### Just because you _can_ automate—_should_ you? - -While integration and automation have the potential to enable scale and save -time and money (we’ve learned this from our -[partners](https:openfn.org/clients)), solutions require investment to set up -and maintain. These costs sometimes outweigh expected efficiency gains and -service outcomes. Therefore, when evaluating the cost-benefit of investing in -automation and integration solutions, we at Open Function Group typically ask 3 -key questions. - -#### 1. Security — Will automation limit the exposure of sensitive data? - -Can the exposure of sensitive data be limited by integrating with secure API -endpoints (rather than relying on human beings to interact with those data, for -example)? Or by automating a data cleaning process? - -#### 2. Accuracy — Will automation increase data accuracy and reduce data entry errors? - -Can the process take place more reliably by limiting the opportunity for human -error (in automating data manipulation or simple algorithmic work, for example)? - -#### 3. Speed — Will automation increase the speed of impact? - -Can the process be done more quickly via automation and is there value in having -it done faster? (The answer to the first part is almost always yes, but -sometimes there's not actually lots of value generated by doing something -faster.) - -#### If you find yourself answering “yes” to these questions, it may be time to consider automating critical processes at your organization. - -Tasks that meet these 3 criteria, take a lot of time to complete or are very -repetitive, and/or involve moving data between apps are typically great -candidates for automation. - -If you find yourself answering “no”, then it may not be worth the investment in -automation... at least not yet. This is especially true if these processes are -still in flux or require a lot of human involvement to complete. That said, now -may be a good time to refine your existing workflows, think about how your -processes might change at scale, and consider what _new_ processes, services, or -outcomes could be unlocked by automation. - -### Delegate your busywork to OpenFn, and try it today! - -If you want to try out automation for your organization, -[sign up](https://www.openfn.org/signup) for OpenFn, free of charge. Check out -[our documentation](https://docs.openfn.org/) and -[website](http://www.openfn.org) to learn how to get started. - -Having trouble setting up your first automation "job"? Email us at -[admin@openfn.org](mailto:admin@openfn.org) for support. Our team is always -happy to assist and help you evaluate the total cost of ownership of automation -solutions. diff --git a/articles/2020-07-02-allow-yourself-to-fail.md b/articles/2020-07-02-allow-yourself-to-fail.md deleted file mode 100644 index dd869d274400..000000000000 --- a/articles/2020-07-02-allow-yourself-to-fail.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -layout: post -title: 'Allow Yourself to Fail' -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [how-to, tips] -featured: false ---- - -Hi all, this is a very short post with a simple message: design for failure. -Even if you've never heard of -[MSSQL](https://www.microsoft.com/en-us/sql-server) (or -[Azure](https://azure.microsoft.com/en-us/), or Microsoft?), I want to talk for -one moment about the importance of upserts and a funny developer term called -"idempotence." - - - -We just extended our -[language-mssql adaptor](https://github.com/OpenFn/language-mssql) with a custom -function that allows upserts (an `upsert` is when you either insert a new record -or update an existing record based on some identifier). Before, you'd need to -write something tedious like: - -```js -sql({ - query: `MERGE my_table AS [Target] - USING (SELECT '8675309' AS some_unique_id, 'writing_blog_posts' AS skill) AS [Source] - ON [Target].some_unique_id = [Source].some_unique_id - WHEN MATCHED THEN - UPDATE SET [Target].some_unique_id=8675309, [Target].skill='writing_blog_posts' - WHEN NOT MATCHED THEN - INSERT (some_unique_id, skill) VALUES ([Source].some_unique_id, [Source].skill);`, -}); -``` - -whereas now you can simply write: - -```js -upsert('my_table', 'some_unique_id', { - some_unique_id: 8675309, - skill: 'writing blog posts', -}); -``` - -For an operation to be idempotent means that it can be repeated time and time -again without producing an unintended result. This is SUPER important for -creating S3 (**S**ecure, **S**table and **S**calable—more on that -[here](https://openfn.org/trust)) integrations because it provides you with two -"get-out-of-jail-free" cards. - -1. If a destination application fails, if a connection times out, or if (for - whatever reason) you're not sure if the `job` was completed (say... making a - payment to CHW) then an idempotent operation can be RETRIED without fear of - making a double-payment. - -2. If you make some change to how your `job` works, make some modification to - one of your destination systems, or just because you want to be _extra extra - sure_ that all the data in a 9 month survey made it to the national public - health reporting system, you can _REPROCESS_ every single message that's come - through OpenFn at the click of a button, without having to worry about - duplicates. - -So... when clients let me mess around with their jobs, I _always_ recommend we -design for idempotence. It's common sense when you're passing messages between -two different systems that are bound to evolve, go offline, have a bad day, etc - -— Taylor - -[Sign up](https://openfn.org/signup){: .btn} to set up a project today, -absolutely free. - -[Reach out](mailto:admin@openfn.org){: .btn} for more information. diff --git a/articles/2020-07-14-cron-is-better-than-a-timer.md b/articles/2020-07-14-cron-is-better-than-a-timer.md deleted file mode 100644 index 687098933f0c..000000000000 --- a/articles/2020-07-14-cron-is-better-than-a-timer.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -layout: post -title: 'Product News: Enhanced Scheduled/Periodic Job Control' -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [annoucement, tips] -featured: false ---- - -Hi all, this is a quick one from the product team at -[OpenFn](https://openfn.org/) — we've made a major upgrade to how timed/period -jobs work. - - - -In the past, if you weren't using OpenFn to drive some real-time (or -"event-based") automation, you'd need to set up an "interval trigger." Like the -photo above, this was essentially a sand timer. Set your trigger to `10` seconds -and your job fetches data from DHIS2, some regional public health data set, or -whatever, then cleans, transforms, and loads it into some other system. - -For the most part, this has got the job done for the last 5 years, but as our -NGO and government clients came up with increasingly specific requirements on -not only how often but _when_ a crucial job gets executed, we began finding -ourselves creating little customizations for them on a once-off basis. We're -happy to annouce that as of `v1.75` (released today), you can now schedule jobs -to run based on `cron` expressions, giving you incredible control over when your -tasks get executed. - -### Scheduling is better than timing. - -Using `cron`, you can choose to run a job every minute by typing `* * * * *`. - -Or maybe you've got a batch sync that you want to take place while your users -are asleep—why not run it every night at 11pm with `23 * * * *`. - -What if you've got to submit reuqests for medical inventory only during the -onset of flu season? Simply type `0 0 1 2-4 *` and your job will run at midnight -the 1st of the month, from February through April. - -You can still run jobs at the click of a button and create timers with -expressions like `*/10 * * * *` for "every 10 minutes", but scheduling with cron -gives OpenFn.org users so much more control over how they run their -organizations. (And that's a good thing.) - -If you're keen on learning by doing but don't have an OpenFn account yet, -[sign up for free](https://www.openfn.org/signup) or mess around with cron -expressions at crontab.guru, -a brilliant site to quickly build complex cron expressions. - -That's all from product for today. Speak soon. - -Taylor diff --git a/articles/2020-12-09-upsert-in-dhis2.md b/articles/2020-12-09-upsert-in-dhis2.md deleted file mode 100644 index 1b2672feadd4..000000000000 --- a/articles/2020-12-09-upsert-in-dhis2.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -layout: post -title: 'Tracked entity instances in DHIS2' -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [how-to, tips] -featured: true ---- - -tl;dr: Lots of our users want to upsert tracked entity instances in dhis2, but -upserts aren’t supported by a standard DHIS2 API endpoint. We built one in our -dhis2 adaptor: it’s composed of existing APIs and a bit of logic 🤔. Now you can -`upsert` tracked entity instances to DHIS2 👍 ✅. - - - -## A bit more... - -An “UPSERT” is a portmanteau of the database functions UPDATE and INSERT. It’s -critical to handle upserts properly when integrating systems. As of version 35 -of the API, DHIS2 does not allow for an administrator to upsert tracked entity -instances (“TEIs”). OpenFn’s own -[Chaiwa Berian](https://github.com/chaiwa-berian) has come up with a solution -that highlights the utility of helper functions in our dhis2 adaptor. By -combining various DHIS2 APIs through an upsertTEI function in OpenFn, DHIS2 -users can now perform upserts to TEIs. - -If you’re curious, check out his implementation -[here](https://github.com/OpenFn/language-dhis2/blob/master/src/Adaptor.js#L347). - -## Even more! - -A tracked entity instance in DHIS2 is a type of entity that can be tracked -through the system. It can be anything from a person to a commodity like a -medicine. If I am a database administrator presiding over two different systems -that are connected to one another, let’s call them “System A” and “System B,” I -would like for any updates made to the TEI of a user named “Jim Smith” in System -A to also appear in Jim’s record in System B. Before upserts came about, doing -so was difficult because of the possibility of duplicate record creation. -Because an upsert simultaneously UPDATES and INSERTS, it prevents duplicates. - -Upserts are important and good because they cut down on the risk of duplicate -data entry and they also allow for transactions to be retried over and over to -ensure data integrity. That last bit is called “idempotency” and you can read -about it [over here](https://blog.openfn.org/allow-yourself-to-fail/). - -Please don’t hesitate to reach out to one of OpenFn’s implementation specialists -if you’d like to learn more. - -— Taylor - -[Sign up](https://openfn.org/signup){: .btn} to set up a project today, -absolutely free. - -[Reach out](mailto:admin@openfn.org){: .btn} for more information. diff --git a/articles/2021-02-03-hosted-or-local-deployment.md b/articles/2021-02-03-hosted-or-local-deployment.md deleted file mode 100644 index bff10a740307..000000000000 --- a/articles/2021-02-03-hosted-or-local-deployment.md +++ /dev/null @@ -1,112 +0,0 @@ ---- -layout: post -title: 'Our Servers or Yours: Thinking through deployment options' -author: Jed Goldstein -author_url: https://github.com/jedbgold -author_image_url: https://avatars.githubusercontent.com/jedbgold -tags: [how-to, tips] -featured: true ---- - -Zandile is a program manager at an iNGO and she needs to use CommCare, DHIS2, -and OpenFn for an upcoming public health project. She understands that all three -pieces of software can be deployed locally, or accessed as SaaS (Software as a -Service). - -Essentially, Zandile needs to decide if she would like to run the software on -someone else’s servers (SaaS), or on her organization’s own servers (deployed -locally). Before making a decision she outlines the basic, non-technical -considerations for both options. - - - -## What is SaaS? - -SaaS is software that is installed and _runs_ on computers maintained by -software professionals, rather than on your own computer. While those computers -might be anywhere in the world, typically you'll access and _use_ this software via -the Internet. - -### Some benefits of SaaS - -With SaaS, the software vendor is responsible for the expenses of managing and -monitoring all of the technical components and issues associated with the -software. This means that Zandile’s iNGO will not be responsible for updating -the software to ensure compliance with new security regulations, maintaining the -servers, backing up the data, purchasing and managing uninterrupted power -supplies, and providing a team of physical security guards to protect the -computers and data therein against physical theft. - -Going the SaaS route is often faster and more secure, because you do not need to -develop expertise in "DevOps" or hire IT and physical security specialists. This -option also provides the greatest amount of flexibility & scalability– because -the SaaS provider is able to deliver more or less computing power, storage, and -bandwidth—right when it’s needed. - -Having smaller setup costs (you don't have to grow a software delivery company -of your own) often makes this a more economical choice for many, though SaaS -will always come with some sort of ongoing fee—a price per month or year that -goes to the vendor to compensate for the time and money they'll spend to ensure -your software works properly. - -## What is Local Deployment? - -Unlike the SaaS option, local deployment means installing and running software -on your own computers—typically on your organization’s servers. - -### Some benefits of Local Deployment - -If a SaaS provider doesn't offer hosting in your country and your government -doesn't allow your data to reside on foreign servers (i.e., you're not allowed -to use things like Gmail, WhatsApp or Facebook for communicating sensitive -information) then local deployment allows you to use tools like CommCare, DHIS2, -and OpenFn while adhering to government data sovereignty regulations. - -Local deployment also provides your organization with complete ownership of the -end-to-end system. Your IT team will be personally responsible for ensuring that -the software works, is maintained, is secure, etc. If your organization does not -already have an IT team in place, then this can become a costly headache, but -for a large organization with embedded IT experience, local deployment often -makes sense. - -Ultimately, being able to directly hire and fire the people who are responsible -for your software's proper functioning can be very useful. It means you have -complete responsibility for whether or not the solution succeeds. - -If you've already got the teams in place (security, DevOps, etc.) then this -option can be more economical in the long run. With a very good DevOps team, -maintaining an extra piece of software might only occupy 20% of a -full-time-employee's salary. For your security guards, if the software is -installed in the same physical location it's possible that your costs won't -increase at all. While there will be very high setup costs, over time you may -realize cost savings by running an efficient software delivery unit within your -organization that spreads its focus around a number of projects. - -## Zandile's Decision - -In this fictional case, data residency is a concern—her data is sensitive or -contains PII—and CommCare, DHIS2 and OpenFn do not provide hosting in the -country she's located. Zandile's organization has a large, experienced IT team -that has managed high-availability software projects for many years... they're -pros. While they anticipate that the setup costs will be quite high (around -$60000 and several months for this set of deployments) they plan on using this -software for the next 5 years and have determined that they'll recoup a -significant portion of that setup cost by not having to pay license fees for -SaaS. They go with local deployment. - -## Which Deployment Option is Best for your organization? - -The answer is: "it depends", but if your organization has never managed local -software deployments, then we recommend going the SaaS approach. SaaS systems, -like the one OpenFn and CommCare offer, are simply going to be more secure, more -stable, and more scalable for the money. - -Crucially, you can always start with SaaS (most tools even offer a free tier) -and then decide later to invest in the big startup costs of a local deployment -if the license fees for the SaaS feel high enough to make local deployment more -economical over the long term. After a few months or years on the SaaS, you'll -likely be in a better position to know if you want to continue using the -software for 5-10 years. - -Should you need any help with your decision though please do not hesitate to -contact OpenFn. diff --git a/articles/2021-02-17-syncing-options.md b/articles/2021-02-17-syncing-options.md deleted file mode 100644 index 61dcc98b3b75..000000000000 --- a/articles/2021-02-17-syncing-options.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -layout: post -title: 'Sync Like You Mean It: Thinking Through System “Syncing” Protocols' -author: Jed Goldstein -author_url: https://github.com/jedbgold -author_image_url: https://avatars.githubusercontent.com/jedbgold -tags: [how-to, tips] -featured: true ---- - -“Syncing” is getting two systems to a state of harmony. This might mean keeping -a list of patients up to date, though modifications can be made in either -system. It might mean copying transactions from one system to another on a -nightly basis. It might mean a lot of things, but the key concept is that when -you sync systems, you’re asking them to work together while simultaneously -respecting both software systems’ independence. - -In this post we’ll discuss two different syncing protocols to consider when -designing your data integration. These include: - -1. **Real-time, or event-based, syncs** -2. **Scheduled syncs** - - -For a -[recent project in Cambodia](/blog/2021/02/09/interoperability_for_case_referrals), -OpenFn is being used by social workers to automate case referrals between the -software systems Primero and OSCaR. In the design phase, we evaluated these two -syncing options. Below, we'll explain what each one is, the differences between -them and which option we chose in the end. - -### Real Time/Event Based Syncs - -The first option considered for this integration was the real-time/event based -sync. This type of sync is triggered whenever a specified event takes place in a -system. With this approach, whenever a case is referred in Primero (via the user -interface, i.e., when a real case-worker clicks the “refer” button) OpenFn -receives a small payload with case data and transmits it to OSCaR and vice -versa. - -![Real_Time_Sync](/img/syncs1.png) - - - -Because of their instantaneous nature, real time/event based syncs are great for -integrations that involve mobile payments or sms messages to recipients. Really, -anything that needs to be done “now”! Additionally, depending on your data -volumes real time syncs might save you money because you’re only using resources -when specific events take place. For instance, in the above example, a run is -triggered by a referral, so if there are only 10 case referrals/month, you'd -only process 10 runs each month. - -This type of sync is great because it’s instantaneous, typically quite -straightforward to set up, doesn’t require any “state mangagement” on OpenFn, -and allows for the reprocessing of individual events. There are, however, -drawbacks. - -For instance, what happens if the app that’s sending notifications to OpenFn -fails to send? What if AWS or GCP goes down, taking half of the internet with -it? If Primero “thinks” it sent the referral, OpenFn never receives it, that -case might not get referred to Oscar! - -### Scheduled Syncs - -![Schedule_Dependent_Sync](/img/syncs2.png) - -The second option considered, a bi-directional schedule dependent sync, solves -for the issue discussed above. On a scheduled basis (every 5 minutes, for -example) OpenFn checks with Primero and Oscar to see if case referrals need to -be transmitted between the two systems and then refers the case if required. In -the unlikely event that any of the software systems involved crash, the -stability provided by the bi-directional sync means that all data is preserved -and eventually makes it to its destination safely. - -The major drawback here is complexity. We had to use 4 jobs instead of 2, and -the job that is responsible for “pulling” data that’s been updated since the -time of the last successful sync has to keep “state”—or some sort of working -memory of what it’s done in the past. When pulling modified cases from Primero, -OpenFn now only pulls cases modified on or after `YYYY-MM-DD HH:MM:SS` where -`YYYY-MM-DD HH:MM:SS` is the time of the last successful, round-trip -synchronization. OpenFn has built-in functionality to handle exactly this -requirement, but not all ETL systems do and it’s a design implication that must -be considered. - -Ultimately, for the project in Cambodia, we decided that this sync option is the -right choice because data integrity is more important than the speed of this -data flow. That’s a crucial point to understand—the organizations operating in -Cambodia decided that for this particular use case, being able to guarantee -eventual syncing was more important than having real-time syncing. - -### Both Sync Options Have Their Pros and Cons - -Both options definitely have their use-cases and OpenFn's platform versatility -enables your team to decide which type of sync is right for your project. - -As always, we are here to help with any questions as you think through which -sync option makes the most sense for your project. diff --git a/articles/2021-05-24-commcare-events.md b/articles/2021-05-24-commcare-events.md deleted file mode 100644 index b94ed1e68879..000000000000 --- a/articles/2021-05-24-commcare-events.md +++ /dev/null @@ -1,383 +0,0 @@ ---- -layout: post -title: 'Forms and Cases: CommCare and event-based integration' -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [how-to, tips] -featured: true ---- - -This is a quick one, but I just got off an exciting call with an organization -that's going to set up some jobs to move data into Salesforce from CommCare and -realized that despite this being one of our more common integration -requirements, we haven't done a 'tips' article for this type of project. Until -now. - - - -So here goes. While this is by no means an exhaustive project planning template, -here are a few things to keep in mind if you're planning to implement a CommCare -to Salesforce integration on your own. - -## Most people use "Data Forwarding" in CommCare - -First, most people make use of CommCare's "Data Forwarding" feature to send form -submissions and changes in cases (creation, update, closure, etc.) to OpenFn in -real-time. You can read about that -[here](/adaptors/commcare#webhook-forward-cases-andor-forms-from-commcare-to-openfn-using-rest-service) -but the key consideration at this planning stage is _when_ you'll be performing -operations—`create(...)`, `update(...)`, `upsert(...)`, `query(...)`, -`(bulk(...)`, etc.—in Salesforce and what data you'll have access to. - -Each time a form submission comes into CommCare, we'll get a copy of that -submission at OpenFn and can use that data to create or modify some records in -Salesforce. - -Likewise, each time a case gets updated (or created or closed) we'll get a copy -of the case with all the case "properties" and we can use that data to _do some -stuff_ in Salesforce. - -If you are using "Form Forwarding", the `trigger` you'd create in OpenFn might -look like this `{"form":{"@name":"ART Adherence Self-Reporting Tool"}}` and it -would trigger your `job` any time an "ART Adherence Self-Reporting Tool" -submission arrived from CommCare, giving that job access to all of the data -inside that submission. - -## Working with the data that comes from CommCare - -Assuming you're using making use of case management, the data that arrives from -CommCare will look something like this: - -```json -{ - "__query_params": { - "app_id": "some-long-id" - }, - "app_id": "some-long-id", - "archived": false, - "attachments": { - "1621866020043.jpg": { - "content_type": "image/jpeg", - "length": 16423, - "url": "https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/1621866020043.jpg" - }, - "form.xml": { - "content_type": "text/xml", - "length": 2727, - "url": "https://www.commcarehq.org/a/your-project/api/form/attachment/some-uuid/form.xml" - } - }, - "build_id": "0ec83881cd0e420dad5c24ed3a5452fe", - "domain": "your-project", - "edited_by_user_id": null, - "edited_on": null, - "form": { - "#type": "data", - "@name": "ART Adherence Self-Reporting Tool", - "@uiVersion": "1", - "@version": "2783", - "@xmlns": "http://openrosa.org/formdesigner/59E1207B-969F-402D-9EEE-675504036F78", - "administrative": { - "coach_verification": "check_here", - "visit_notes": "", - "vist_notes_to_save": "" - }, - "case": { - "@case_id": "1ec51ee9-5aef-4bd2-b7eb-7599856251bc", - "@date_modified": "2021-05-24T14:20:28.693000Z", - "@user_id": "332e893dcd1b413686621bd80aae0cd3", - "@xmlns": "http://commcarehq.org/case/transaction/v2", - "update": { - "consent_received": "yes", - "home_visit_notes": "" - } - }, - "meta": { - "@xmlns": "http://openrosa.org/jr/xforms", - "appVersion": "CommCare Android, version \"2.51.2\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17", - "app_build_version": 2798, - "commcare_version": "2.51.2", - "deviceID": "commcare_a39f55a5-c744-4e33-8e01-d17e7698894f", - "drift": "0", - "geo_point": null, - "instanceID": "130c68c5-7d17-4086-8a85-27d7d7da2216", - "timeEnd": "2021-05-24T14:20:28.693000Z", - "timeStart": "2021-05-24T14:18:46.856000Z", - "userID": "332e893dcd1b413686621bd80aae0cd3", - "username": "some-chw" - }, - "participant_information": { - "participant_id": "007", - "name": "taylor downs", - "gender": "male", - "guardian_information": { - "guardians_name": "Fake Data", - "guardians_phone_number": "8675309", - "guardians_signature": "1621866020043.jpg", - "relationship_to_participant": "father" - }, - "current_medications": [ - { "name": "generic-1", "active": true }, - { "name": "fakelyn-notrealiol", "active": false }, - { "name": "sasstra-zenica", "active": false }, - { "name": "ibuprofen", "active": true } - ] - }, - "tested_for_hiv_status_tested_for_hiv": "OK", - "visit_information": { - "consent_given": "yes", - "date_consent_given": "2021-05-23", - "visit_date": "2021-05-23" - } - }, - "id": "130c68c5-7d17-4086-8a85-27d7d7da2216", - "indexed_on": "2021-05-24T14:20:39.045971", - "initial_processing_complete": true, - "is_phone_submission": true, - "metadata": { - "appVersion": "CommCare Android, version \"2.51.2\"(463994). App v2798. CommCare Version 2.51.2. Build 463994, built on: 2021-03-17", - "app_build_version": 2798, - "commcare_version": "2.51.2", - "deviceID": "commcare_a39f55a5-c744-4e33-8e01-d17e7698894f", - "drift": "0", - "geo_point": null, - "instanceID": "130c68c5-7d17-4086-8a85-27d7d7da2216", - "location": null, - "timeEnd": "2021-05-24T14:20:28.693000Z", - "timeStart": "2021-05-24T14:18:46.856000Z", - "userID": "332e893dcd1b413686621bd80aae0cd3", - "username": "some-chw" - }, - "problem": null, - "received_on": "2021-05-24T14:20:37.976363Z", - "resource_uri": "", - "server_modified_on": "2021-05-24T14:20:38.111789Z", - "type": "data", - "uiversion": "1", - "version": "2783" -} -``` - -This is a big blob of `JSON`—the body of the message that's received at OpenFn -when this particular form ("ART Adherence Self-Reporting Tool") is submitted in -CommCare—will be handed off to the job to start processing. The question is, -what should we do? - -When setting up for a self-service implementation on OpenFn, the most important -thing you can do at this moment is carefully enumerate the data entry process -that you'd like a real human to follow. You can translate it to a job script -later. - -You'll need to write this up for your own case, but in this fictional example, -here's the data entry process. - -## The instructions for our worker - -:::tip - -Right from the start, notice that we're being incredibly explicit with these -instructions! We're using the "API Name" (instead of just the "label", which -might be ambiguous) of every field we want filled out in Salesforce and we're -using the specific "path" to the data we want this person to enter from -CommCare. - -**Why are we being so specific?** Because eventually, a computer will need to -interpret this—and they're _terrible_ with ambiguity! - -::: - -1. Every time a messaged is received with - `{"form":{"@name":"ART Adherence Self-Reporting Tool"}}` in the body (this is - our trigger) -2. Log into Salesforce and create a new participant with the `participant_id` - you find in the `form.participant_information` section as their - `Participant_Code__c`. (If one already exists in Salesforce with that code, - then update the existing record instead.) -3. Fill out the following fields in Salesforce based on the CommCare data in - this message: - - `Name__c` with the data from `form.participant_information.name` - - `Sex__c` with the data from `form.participant_information.gender` - - `CommCare_Case_ID__c` with the data from `form.case.@case_id` -4. After you've created (or updated) this participant in Salesforce, create a - record of the visit with the `instanceID` from the `metadata` section as the - unique identifier `Visit_Code__c`. (Again, if there's already a visit with - that ID please update the existing record.) -5. Fill out the following fields for the visit with data from CommCare" - - `Date__c` with `form.visit_information.visit_date`. - - `Consented__c` with `form.visit_information.consent_given`. - - Always set `Test_Status__c` to `true`, regardless of what's in the message - from CommCare. - - And relate this record with the `Community_Health_Worker` by their username - in `form.metadata.username`. -6. Finally, add a record for each medication listed in the - `form.participant_information.current_medications` array—matching on a unique - ID formed by a combination of the medication `name` and the `participant_id` - so that we can update existing medication records if they're present. -7. Fill out the following fields for the medication: - - `Generic_Name__c` with `name` - - `Status__c` with `active` - - And relate this record with the participant you created or updated in step - 2 via the `participant_id` field. - -Phew... that's the task. It's just a fictional example and things could be much -more straightforward, or much more complicated than this, but it's important to -remember that if you can get to this level of **precision and granularity** in -your data entry process, a tool like OpenFn can automate this for you in a -flash. - -## Translating this into an OpenFn project - -If you're streaming data in from CommCare and you've got your Salesforce system -all set up so that this data entry person can complete the above steps (are all -the objects and fields created? are the right fields marked as "unique" and set -to be used as an "external id" in the Salesforce administration section? have -you turned on data forwarding in CommCare?) then it's time to turn them into an -OpenFn project! - -:::tip - -A quick plug: **Did you know that there's an -[OpenFn community forum](https://community.openfn.org)** where you can post -stuff like the "steps" above and get help from other OpenFn users and staff -converting these steps into a real, working, OpenFn job? - -Well, you do know! Check it out at -[community.openfn.org](https://community.openfn.org) - -::: - -### Create a Salesforce credential - -We don't need a CommCare credential, since they'll send data to us. Create a -Salesforce credential that will allow the OpenFn worker to log into your -Salesforce system. - -Read more about credentials [here](/documentation/build/credentials). - -### Create a message-filter trigger - -- Select `Message Filter` for the `type` -- Enter `{"form":{"@name":"ART Adherence Self-Reporting Tool"}}` for the - `inclusion criteria` - -Read more about triggers [here](/documentation/build/triggers). - -### Create the job - -- Give it a name -- Select the trigger you just created -- Select the `salesforce` adaptor -- Select the credential you just created - -And convert the instructions above to "operations" by using the inline help -provided by the Salesforce adaptor: - -```js -// Use upsert to create or update a participant based on their participant code. -upsert( - 'Participant__c', - 'Participant_Code__c', - fields( - field( - 'Participant_Code__c', - dataValue('form.participant_information.participant_id') - ), - field('Name__c', dataValue('form.participant_information.name')), - field('Sex__c', dataValue('form.participant_information.gender')), - field('CommCare_Case_ID__c', dataValue('form.case[@case_id]')) - ) -); - -// Then upsert a visit using the visit code. -upsert( - 'Visit__c', - 'Visit_Code__c', - fields( - field('Visit_Code__c', dataValue('metadata.instanceID')), - field('Date__c', dataValue('form.visit_information.visit_date')), - field('Consented__c', dataValue('form.visit_information.consent_given')), - // Always set status to true - field('Test_Status__c', true), - // And related this visit to the participant we just created by their "code" - relationship( - 'Participant__r', - 'Participant_Code__c', - dataValue('form.participant_information.participant_id') - ) - ) -); - -// And finally for EACH mediation listed, create a medication record with a status -each( - merge( - dataPath('form.participant_information.current_medications[*]'), - fields( - field('pID', dataValue('form.participant_information.participant_id')) - ) - ), - upsert( - 'Medication_Tx__c', - 'Medication_Tx_ID__c', - fields( - field(Medication_Tx_ID__c, state => { - // Here, inside the medications array we've "scoped" state so that - // state.data, for each item in the array, looks like this: - // { pID: 007, name: "sasstra-zenica", active: false } - - // We will concatenate the participant ID with the medication name. - return state.data.pID + state.data.name; - }), - field('Generic_Name__c', dataValue('name')), - field('Status__c', dataValue('status')), - relationship('Participant__r', 'Participant_Code__c', dataValue('pID')) - ) - ) -); -``` - -Now, every time this job runs (which is every time a CommCare form is submitted) -your OpenFn worker will upsert a `Participant`, upsert a `Visit`, and upsert a -whole list of `Medications` in Salesforce. - -## What's next - -Well, in our little example you'd turn the job "on" (setting it to on the -inbound messages from CommCare) and let it run. Whenever there was a failure -(maybe your Salesforce admin added a new required field on the -custom`Medication` object) you'd get an email and you'd have to come back to -OpenFn to update your job, including that new field. - -If you're in the process of designing your CommCare and Salesforce systems at -the moment, this back-and-forth will be pretty common. Keep in mind that you -want as much simplicity as possible in those end-user systems because... well -because _humans_ have the interact with them every day! - -So long as your processes are well defined, OpenFn can handle a bit of -complexity (data cleaning, transformation, complex logical flows, etc.) but you -should never make sacrifices to the user experience in CommCare and -Salesforce—that's a quick way to lose adoption. - -So, ideally, you've designed your workflows in CommCare and Salesforce to make -your users happy and get them the information they need to do their jobs well -and _then_ you come back to OpenFn and spell out our data entry instructions -like we've done above. - -## A final thought - -The two most important resources you've got at your disposal if you're setting -this all up on your own are: - -1. this site (docs.openfn.org), and -2. the [forum](https://community.openfn.org) (community.openfn.org) - -Read through the -["What is an integration"](/documentation/tutorials/tutorial), -["OpenFn Concepts"](/documentation/get-started/terminology), and -["Build"](/documentation/build/workflows) sections if you're a thorough, -background-first kind of learner. If you crave snippets and sample job code, -head directly to the [Job Library](/adaptors/library) to see how other OpenFn -users are creating their jobs. - -Either way, keep the community posted on your progress in the forum—you'll find -lots of helpful folks willing to lend you a hand in your integration journey. diff --git a/articles/2021-07-05-wrapping-my-head-around-jobs.md b/articles/2021-07-05-wrapping-my-head-around-jobs.md deleted file mode 100644 index 65ced6180e76..000000000000 --- a/articles/2021-07-05-wrapping-my-head-around-jobs.md +++ /dev/null @@ -1,300 +0,0 @@ ---- -layout: post -title: Wrapping my head around jobs -author: Taylor Downs -author_url: https://github.com/taylordowns2000 -author_image_url: https://avatars.githubusercontent.com/taylordowns2000 -tags: [how-to, tips, jobs] -featured: true ---- - -Jobs are business processes turned into functional-style scripts. What does that -mean, how should you approach writing jobs? - - - -First, this is how _I_ think about jobs and what we do at Open Function Group to -try to make our job code as readable, future-proof, and concise as possible. -There are a million different ways to approach writing jobs. This is one. - -## It all starts with `state` - -If a job is a set of instructions for a chef (a recipe?) then the initial -`state` is all of the ingredients they need tied up in a perfect little bundle. -It usually looks something like this: - -```json -{ - "configuration": { - "hostUrl": "https://moh.kenya.gov.ke/dhis2", - "username": "taylor", - "password": "very-secret" - }, - "data": { - "type": "registration", - "patient": { - "age": 24, - "gender": "M", - "nationalId": "321cs7" - } - } -} -``` - -This might be the initial `state` for a real-time, message-triggered job. Some -source system generated a new patient payload and sent that payload to OpenFn. -The data from our source system will wind up in `state.data`. Now if my job is -meant to take this new patient registration information and use it to create a -new record in the national health record system, I'll also need to provide my -robot-chef here with a credential so they can access that system. The credential -I've specified will get put into `state.configuration` and now our "raw -ingredients" are all ready for our robot chef. - -Note that even if this job was initiated by a cron trigger (e.g., "Hey chef, -prepare this recipe every Tuesday at 7pm") or by a flow/catch trigger (e.g., -"Hey chef, prepare this recipe only when you _fail_ to make banana pancakes") it -will have an initial state. - -**Every job, and every operation inside that job (think "step" in a recipe) is -called with `state` and returns `state` when it's done.** - -Initial state for a cron triggered job might look like this: - -```json -{ - "configuration": { - "hostUrl": "https://moh.kenya.gov.ke", - "apiKey": "abc123" - }, - "data": {}, - "lastProcessedId": 321 -} -``` - -And for a fail triggered job like this: - -```json -{ - "configuration": { - "hostUrl": "https://moh.kenya.gov.ke", - "apiKey": "abc123" - }, - "data": {}, - "lastProcessedId": 321, - "error": ["Required field missing", "Patient Surname", "Line 43"] -} -``` - -No matter what, jobs start with state. See -["Initial and final state for runs"](/documentation/jobs/state) for a detailed -breakdown. - -## It ends with `state` too - -Now that we've got it in our heads that `state` is the raw ingredients you hand -to your chef when you ask them to prepare a recipe, let's look at the recipe. -Boiled down (excuse the pun) a job for loading those patients into the national -health record system might look like this: - -```js -get('/api/insuranceRegistrations'); -post('/api/patients', { ...someData }); -post('/api/visits', { ...someData }); -``` - -We're telling our chef to take those raw ingredients (login info for our -national health system and a chunk of information about a newly registered -patient) and do the following: - -1. Find out whether this person already has a national health insurance number -2. Add this person to the patient registry (making use of some insurance data - from step 1) -3. Add a visit record with information about this initial visit (making use of - patient registry data from step 2) - -When all of this is done, we'll not only have a new patient and visit logged in -the national health registry, but we'll also return a final `state` object with -information about what we've done that can be used in subsequent jobs. Imagine -that we want to make a cash transfer to this patient so that they can take a cab -to the next visit—we might create a job with the Mpesa adaptor that takes the -final state of this first job as its _initial state_. In this way, jobs are -composable. - -But what about the complexity inside our job—in order to complete step 2, we -need some data from the insurance registry and we only get that data in step 1. -Crucially, each operation (again, think "step" in a recipe) takes state and -returns state. In effect, the OpenFn execution pipeline simply calls all of your -action methods _with state_, passing it along from one operation to the next, -waiting for each to finish and using the output from the first as the input for -the second. - -While you may write your `get`, `post`, `post` job as it's show above, the way -it's handled by OpenFn is actually more like: - -```js -return get('/api/insurance', { ...useDataFromState })(state) - .then(state2 => post('/api/patients', { ...useDataFromState2 })(state2)) - .then(state3 => post('/api/visits', { ...useDataFromState3 })(state3)); -``` - -Each of these operations returns a function which _takes state_ and returns -state. This means that _within_ a job, you are essentially modifying `state`, -creating/manipulating records in external systems, and returning `state`. - -It opens up a really interesting world of possibility for data manipulation, -cleaning, or transformation. Consider what we might do _after_ we get data from -the insurance registry but _before_ we create that patient in the national -patient registry: - -```js -get('/api/insuranceRegistrations'); -fn(state => { - console.log(state.data); // let's look at the response from the insurance API. - state.data.people.filter(p => p.HasActiveInsurance); // and modify the payload to only retain those with active insurance - return state; // before returning state for our create patients operation. -}); -post('/api/patients', { ...someData }); -post('/api/visits', { ...someData }); -``` - -We might even need to do some manipulation _before_ we send a `get` request to -the insurance registry. That's no problem: - -```js -fn(state => { - state.data.registrationType = state.data.age > 18 ? 'Adult' : 'Minor'; - return state; // before returning state for our create patients operation. -}); -get('/api/insuranceRegistrations', { - query: { type: dataValue('registrationType') }, -}); -fn(state => { - state.data.people.filter(p => p.HasActiveInsurance); - return state; -}); -post('/api/patients', { ...someData }); -post('/api/visits', { ...someData }); -``` - -Here, we've added a step to modify the initial `state` before we send that first -`get` request to the insurance API. We determine if the new patient is a minor, -and then use that newly calculated data to apply a query to the insurance API -request. - -Using `fn(state => state)` or `alterState(state => state})` is incredibly -useful, because it allows us to separate our data manipulation, calculation, and -raw Javascript (which will be harder for low-tech users to understand) from our -external actions. Let's explore that some more. - -## Keeping external actions clean - -Inside each operation we could do some data manipulation... all of these -operations, across the many different language packages, allow for inline data -manipulation like this: - -```js -get('/api/insuranceRegistrations', { - query: state => { - console.log("I'm doing some fancy stuff here."); - return { type: state.data.age > 18 ? 'Adult' : 'Minor' }; - }, -}); -post('/api/patients', { - body: { - name: state => { - return `${state.data.firstName}${state.data.lastName}`; - }, - }, -}); -``` - -But if you're interacting with both technical and non-technical users, it makes -for harder to read jobs. Consider the following instead: - -```js -// Perform calculations... -fn(state => { - // Create several new calculated attributes... - state.data = { - ...state.data, - type: state.data.age > 18 ? 'Adult' : 'Minor', - fullName: `${state.data.firstName}${state.data.lastName}`, - }; - - return state; -}); - -// Get insurance data... -get('/api/insuranceRegistrations', { query: { type: dataValue('type') } }); - -// Create new patient... -post('/api/patients', { body: { name: dataValue('fullName') } }); -``` - -Since we often have non-developers creating the external operations like `get` -and `post` above, this pattern makes our handoff easier. The business analyst -can say "I need to have a registration `type` field available for use when -querying the insurance registry." A developer might respond, "Great! How do you -want to calculate it... I've got all of Javascript at my fingertips." That dev -can then make as many API calls as they'd like, perform as many -`map.reduce(...)` calls as their heart desires to complete that calculation... -so long as they make sure the hand off `state` to the business analyst's -operation with a valid `state.data.type` attribute. - -A final benefit of this approach is that it becomes much easier to generate job -scripts from Google Sheets. Our implementation team frequently works with -non-technical clients to generate field maps that look like this: - -| Path to Source Data | Destination Field | Auto-generated syntax (using concat) | -| ------------------- | ----------------- | --------------------------------------------: | -| patient.fullName | name | field('name', dataValue('patient.fullName')), | -| patient.age | age | field('age', dataValue('patient.age')), | -| ??? | type | plz help us calculate 'type' based on x, y, z | -| patient.sex | gender | field('gender', dataValue('patient.sex')), | - -We can then copy and paste the syntax generated in that final column directly -into OpenFn and update the bits that need some sort of custom code, writing an -`fn(state)` block or an `alterState(state)` block before the external action. - -## Wrapping up - -Some key takeaways here: - -1. Jobs start and end with `state` — some raw ingredients that will be used in a - recipe. - -2. Jobs are lists of `operations` — steps in a recipe that _each_ take `state`, - _do some stuff_, and then return `state`. - -3. As you move through the steps in a job, you are modifying `state`. Each - subsequent step begins with the final state from the previous step. - -4. It may be useful to keep all your custom Javascript data cleaning, - manipulation, etc., in a separate operation (e.g., `fn(state)` or - `alterState(state)`) so that your external actions are clean and easy to - follow. - -Finally, taking a close look at how developers write those `fn(state)` steps -tells us a lot about what the job execution pipeline is really doing: - -```js -// here, "fn" is a function that takes state and returns state -fn(state => { - console.log("I'm doing some cool stuff."); - // I might create some new attribute... - state.myNewThing = true; - - // And ALWAYS return state for the next operation to use... - return state; -}); -``` - -I hope this gives you sense of how I think about structuring jobs and building -data pipelines or automation flows on OpenFn. We recognize that this stuff is -complex, and are pushing our new documentation regularly, so please do get in -touch if you think there are ways we could improve this type of -walk-through/helper article. - -Happy integrating, - -Taylor diff --git a/articles/2021-10-08-improving-multistage-docker-builds-using-buildx.md b/articles/2021-10-08-improving-multistage-docker-builds-using-buildx.md deleted file mode 100644 index 62f193b43c62..000000000000 --- a/articles/2021-10-08-improving-multistage-docker-builds-using-buildx.md +++ /dev/null @@ -1,207 +0,0 @@ ---- -layout: post -title: Improving Multistage Docker Builds using Buildx -author: Stuart Corbishley -author_url: https://github.com/stuartc -author_image_url: https://avatars.githubusercontent.com/stuartc -tags: [how-to, docker, ci/cd] -featured: true ---- - -So you're using docker's multi-stage builds and noticed that your build times -aren't nearly as quick as you expected? - - - -As many teams who spend more and more time using docker, it's quite common to -get into multi-stage builds; usually resulting in significantly smaller images. - -However this comes with a pretty significant dilemma with caching. Even when -using the `--cache-from` flag when building, docker only caches the last image. - -One proposed solution[1](#ref1), is to pull, build and push each -individual stage. Coming with tight coupling between the shape of your -Dockerfile and your build process/scripts. - -The other solution uses Docker Buildx which the document describes as: - -> Docker Buildx is a CLI plugin that extends the docker command with the full -> support of the features provided by Moby BuildKit builder toolkit. It provides -> the same user experience as docker build with many new features like creating -> scoped builder instances and building against multiple nodes concurrently. - -While that sounds pretty cool, it doesn't really touch on caching. This actually -took me a while to find out that it would in fact do caching very differently. -In fact it's a very different experience using it, and has lots of really cool -features that further detach you from the local docker state allowing you to -build in environments that are stateless - such as Google CloudBuild without -having to wire up some kind of persistence or file caching scheme. - -## Buildx - -We're only going to scratch the surface of Buildx, and with that let's get the -absolute minimum working; build our image locally. - -### Local Cache - -First things first we need to create a builder, and select it for use. This is -important as without creating a buildx builder (and setting it as the default), -buildx will use the `docker` driver instead of the `docker-container` driver -which we want in order to take advantage of cache exporting. - -``` -docker buildx create --name mybuilder --use -``` - -> You only need to run this once, except in the case of CloudBuild where each -> invocation is a new node. - -``` -docker buildx build \ - --cache-from=type=local,src=/tmp/buildx-cache \ - --cache-to=type=local,dest=/tmp/buildx-cache \ - --load \ - . -``` - -While the `--cache-*` options aren't specifically required when running `build`, -as `buildx` does manage its own local cache (distinct from the regular docker -cache), it's there to emphasise the options that cache can be provided via the -CLI options. - -This is about as close as you get to a regular docker build, with the -significant difference being that you have to specify where to cache from and -to. - -The `--load` flag is to tell buildx to set the output to the local docker -daemon. Without that you won't actually get a resulting image to run. However, -depending on your use case, this could be seen as a convenience - if you're -wanting to run your tests inside your build; a resulting image isn't -particularly useful. - -### Remote Cache - -Now comes to the part I'm most interested in, caching in a stateless/remote -environment. Multipart builds for us at OpenFn are essential, since we use -Elixir and like other compiled languages there is a lot to be gained by only -shipping the stuff you're going to run; and no language is safe from requiring -several times more 'stuff' in order to build our apps. - -Buildx supports a -[handful of different types](https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#-export-build-cache-to-an-external-cache-destination---cache-to) -of caching sources and destinations. We're going to be using the `registry` -type, where you point the cache at a repository reference (repo/image:tag -style). - -> One thing to note is that Google Container Registry does not support the -> metadata/manifest format that buildx uses, so if you're using Google Cloud you -> will need to start using Artifact Registry. - -**Inline** - -Push the image and the cache together: - -``` -... ---cache-from=type=registry,ref=$IMAGE_NAME \ ---cache-to=type=inline \ -... -``` - -This comes with the constraint that cache mode is always `min`, which only -exports/caches the resulting layers; which is still better than the plain docker -build caching but I think having the intermediary layers is generally a win. We -want to avoid a single line change invalidating an entire build step. - -**Registry** - -Resulting image and cache are separated: - -``` -... ---cache-from=type=registry,ref=$IMAGE_NAME-build-cache \ ---cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \ -... -``` - -Again coming back to the cache mode, here being `max`; all intermediary laters -are exported to the cache image as well. - -I have opted to create _two_ images, one for caching and another for the -resulting image used to deploy. This gains us a much more granular cache and the -ability to more easily manage the cache image - like deleting the whole thing -when wanting to invalidate the cache. Not to mention I'm fairly sure the size of -our images that get pulled on kubernetes would get significantly larger with -many more layers. - -It feels like a safer bet to have lean images for kubernetes to pull, and chunky -cache images specifically for speeding up build. - -Depending on your setup, pulling large images can get _seriously_ expensive in a -reasonably active deployment environment - like on AWS ECS without using -PrivateLink. - -> It appears the `moby/buildkit` documentation also demonstrates -> [this](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) -> approach. - -``` -IMAGE_NAME=us-east4-docker.pkg.dev//platform/app \ -docker buildx build \ - -t $IMAGE_NAME:latest \ - --cache-from=type=registry,ref=$IMAGE_NAME-build-cache \ - --cache-to=type=registry,ref=$IMAGE_NAME-build-cache,mode=max \ - --push \ - --progress=plain \ - . -``` - -This implies that the cache image is named with the suffix `-build-cache`: -`us-east4-docker.pkg.dev//platform/app[-build-cache]`. - -The `--push` argument tells buildx to push the resulting image to the registry. - -## Tips - -**Clearing the local cache** - -As mentioned before, buildx has its own cache and in order to clear the cache -while debugging and readying a Dockerfile for remote building you'll probably -need to reach for `docker buildx prune`. - -## Closing thoughts - -Using buildx has been a really pleasant experience, having personally attempted -using it a few times over the last 3 years; the most recent one being the first -time I felt confident getting it into production. As with any sufficiently -flexible build tooling, the errors and issues you can run into range from -complete gibberish, genuinely concerning inconsistencies to architectural -choices that you haven't fully caught up on; requiring an ever growing list of -changes you need to make to your own build process. - -Our initial observations have been great, reasonable changes on our build have -gone from 28 minutes to around 9 minutes. - -While I have encountered a few confusing cache invalidations, especially when -building locally, exporting the cache to a repository and then having CloudBuild -use the image cache. And occasionally locally having what feels like _really_ -aggressive caching on intermediate steps, leading me to pruning the local cache. - -But overall, these issues aren't necessarily buildx issues and more likely a -combination of building docker images in general except with many more steps -accounted for by the cache. - -It's kinda hard to see now what the exact issues I had with it in the past, but -hey! - -Buildx has given me what I 'expected' with docker multi-stage builds, and having -the cache in a repository completely side-steps having to attach a shared volume -or copying from a storage bucket. - -## Resources - -- [Multi-stage builds #3: Speeding up your builds](https://pythonspeed.com/articles/faster-multi-stage-builds/) - 1 -- [Docker Buildx](https://docs.docker.com/buildx/working-with-buildx/) -- [buildx build reference](https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#buildx-build) -- [mody/buildkey Registry cache exporter](https://github.com/moby/buildkit#registry-push-image-and-cache-separately) diff --git a/articles/2021-10-15-webpack-to-esbuild-part1.md b/articles/2021-10-15-webpack-to-esbuild-part1.md deleted file mode 100644 index 2dcd7ce45829..000000000000 --- a/articles/2021-10-15-webpack-to-esbuild-part1.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -layout: post -title: Moving from Webpack to esbuild on Phoenix -author: Stuart Corbishley -author_url: https://github.com/stuartc -author_image_url: https://avatars.githubusercontent.com/stuartc -tags: [how-to, js, webpack, build, phoenix, ci/cd] -featured: true ---- - -We're very happy users of Elixir and Phoenix at OpenFn, given that we've been -using it continuously for about 6 years - upgrades and all. Our front-end -toolchain, albeit far from out of date (Webpack `5.52.1` today) has left some -room for improvement. - - - -> This is a post written on what we call 'Slow Fridays', where we explore and -> think about stuff we're curious about. The deal is some artifact at the end of -> the day. So while this post is not complete - I believe part two is warranted. - -Phoenix 1.6 started to include esbuild by [default](#ref1). The main reasons -cited were the amount of support the team had to continuously give for a -toolchain that although necessary is not under the control of the framework -teams control. - -Back in the day, Phoenix used brunch to do front end builds - despite not being -the most common bundler at the time. The objective I assume was to provide -something that is easy to get going with and lowers the barrier to entry for -Phoenix. We actually switched brunch out for webpack before webpack got into -Phoenix in order to more easily use new libraries and newer ES syntax. - -It wasn't without its challenges, and still is. Webpack is wonderfully -powerful - but it does do _a lot_, and it's really easy to stumble onto plugins -that are either order dependent or mutually exclusive. - -## So what's this about esbuild - -It's a bundler/compiler for JS/TS projects that is written in Go, along with -being faster by the simple fact it's natively compiled it also leverages -concurrency. To compare _why_ esbuild is faster than NodeJS on face value is -unfair in my opinion. Although having similar objectives, they are wildly -different in implementation and in features. - -So **speed** is the big selling point. But thinking back to other bundlers, what -made us switch (thinking of gulp, grunt, mixing in babel, browserify etc). Those -changes were never about speed, at least a drop in build time was nice - a bit of -caching goes a long way. The changes were about being able to use the syntax and -libraries we wanted with as little fuss as possible. - -Webpack _can_ do almost anything. I'm not convinced esbuild can match that, and -as an open-source maintainer I'd argue it shouldn't break its original goals to -match Webpack. - -## Exploring the caveats - -I started breaking down what our current bundle setup actually does, after all -if we were just bundling plain JS I'd probably be using rollup and terser. Going -through our `webpack.config.js` file I can see that it: - -- Conditionally provide sourcemaps depending on the environment. -- Uses babel to parse js/jsx files - At the same time cherry picks lodash imports - *. -- Parse `import`s for `.css` files through `style-loader` and `css-loader` - Injecting the styles onto the dom. -- Detect `require` statements to images, copy them to an assets folder - and replace the statement with a url to image. -- Can watch files (excluding `node_modules`) in development. -- Splits files that come from `node_modules` into a `vendor.js` bundle. - -That's a lot more than just building something, there's some implicit behaviour -here. - -- The interaction between `style-loader` and `css-loader` results in extra - functions and behaviour being introduced, it's not producing a `.css` file. - > I'm personally not a big fan of apps injecting styles, but I think I get why - > people do it. Maybe I'm old-school and like to have my stylesheets delivered - > in a few files (or even one). -- The splitting doesn't know about `app.js`, it puts everything that resolves to - `node_modules` in `vendor.js`. Subtle but worth pointing out. - -* -Could probably ignore this and refactor some -files and check that tree-shaking is working properly. - - -## What we need esbuild to provide - -Like I mentioned before, webpack is super versatile and it would be -short-sighted to get esbuild to do everything it does. - -So I'm approaching this with the idea that I code will need some changes, the -less the better - I've always firmly believed the kinds of restrictions smaller -tools provide can make things neater and more portable. - -- ✅ **Provide source maps** - Supported out of the box. [docs](https://esbuild.github.io/api/#sourcemap) -- ✅ **Watching files** - Yup. [docs](https://esbuild.github.io/api/#watch) -- 🆗 **Build our css** (and vendored css) - Compiling CSS is supported, however it doesn't (as far as I can tell) doesn't - replace `style-loader` and how it injects CSS into the DOM. -- ✅ **Copy image assets referenced in JS** - Indeed it can, using the `file` loader. - [docs](https://esbuild.github.io/content-types/#external-file) -- ✅ **Can split our files** - I picked a confusing green checkbox to illustrate that while esbuild does - support file splitting - it appears to have some caveats, primarily that it - only works with `esm` output files. We don't ship ES Modules to the browser, - but seems like a good moment to try given the primary targets for our web - front end getting module support in 2019 2. - -## Giving it a go - -I've tried to go over the process as methodically as possible, after all this -isn't a new codebase nor am I alone on this. It's always important to understand -who's going to be working with this, and respect the varying skill sets and -focuses of our peers. - -So we know what we want, but can any of this work? Let's give it a go with the -simplest of steps: - -``` -./node_modules/.bin/esbuild js/app.js --bundle --outfile=out.js - > js/app.js:58:2: error: Unexpected "<" - 58 │ - ╵ ^ - -1 error -``` - -Bang! Right, that one makes sense to me, it doesn't know what JSX is yet. I'm -gonna take a wild guess and say it won't know about the CSS or our referenced -images. - -``` -./node_modules/.bin/esbuild js/app.js \ - --loader:.js=jsx \ - --loader:.png=file \ - --loader:.jpg=file \ - --loader:.jpeg=file \ - --bundle \ - --outdir=out - > js/marketing/ProductsList.js:53:40: error: Unexpected ":" - 53 │ const displayList = products.filter(::this.includesText); - ╵ ^ - - > js/marketing/Banner.js:13:4: warning: Duplicate key "background" in object literal - 13 │ background: `-webkit-linear-gradient(to left, ${theme.palette.banner.right}, ${theme.palette.banner.left})`, - ╵ ~~~~~~~~~~ -... -``` - -Nice! Two things worth mentioning here: - -1. We're using some non-standard ES syntax here: `::`, let's replace that with - the equivalent `includesText.bind(this)`. -2. Some duplicate keys in our theme objects, looks like a warning but worth - cleaning up. - -``` -./node_modules/.bin/esbuild js/app.js \ - --loader:.js=jsx \ - --loader:.png=file \ - --loader:.jpg=file \ - --loader:.jpeg=file \ - --bundle \ - --outdir=out - - out/app.js 8.0mb ⚠️ - out/commcare-dhis2-JKJLM3MQ.png 471.0kb - ... - out/app.css 85.1kb - ...and 145 more output files... - -⚡ Done in 651ms -``` - -Jeepers! 651ms! That's nuts. - -We can see it's copied our images and css into the build folder. Note that we're -not doing any bundle splitting right now and from the looks of it - -``` -... \ -> --minify - - out/app.js 3.4mb ⚠️ - ... -⚡ Done in 611ms -``` - -That's better, wow. It's kinda difficult to not be amazed. For context, a -minified and split production build takes about 34s with webpack and that's on my -i7 desktop machine, and 197s (3+ mins) on CI/CD. - -## What's next? - -So our 'can we actually do this' seems to have gone pretty well so far. I'm -really excited about what this will give us in the end. - -But a shell command doth not a replacement for webpack make. We still need to: - -1. Get Phoenix to use esbuild and watch our files as we work. -2. Ensure that our html templates serve the correct files in dev & production. - Including our CSS that is no longer injected into the DOM. -3. Split at least our vendored modules into their own bundle. -4. Make sure sourcemaps generate correctly for when we upload them to Sentry. -5. Have some kind of cache-busting naming scheme for production builds. - -## Resources - -- [FYI: Phoenix drops webpack and npm for esbuild](https://fly.io/blog/phoenix-moves-to-esbuild-for-assets/) 1 -- [JavaScript modules via script tag](https://caniuse.com/es6-module) 2 diff --git a/articles/2021-10-22-testing-react-app-with-jest-hound.md b/articles/2021-10-22-testing-react-app-with-jest-hound.md deleted file mode 100644 index 44d6e26c19b8..000000000000 --- a/articles/2021-10-22-testing-react-app-with-jest-hound.md +++ /dev/null @@ -1,290 +0,0 @@ ---- -layout: post -title: Testing a React app, the blurred line between Unit, integration and E2E -author: Chaiwa Berian -author_url: https://github.com/chaiwa-berian -author_image_url: https://avatars.githubusercontent.com/u/7937584?v=4 -tags: [how-to, tips, testing, browser-testing, react, elixir] - -featured: true ---- - -Have you ever struggled to layout the strategy for testing your React App? Well, -you are not alone! Here a few hints from the lessons I learned during my -experience testing a -[React](https://reactjs.org/)/[Redux](https://redux.js.org/) app with a -[Phoenix](https://www.phoenixframework.org/)/[Elixir](https://elixir-lang.org/) -backend. - - - -## The Blurred Line - -Because a React app is built on -[components](https://reactjs.org/docs/react-component.html), the basic UI units, -it is natural to think and organise your tests around components! And so unit -testing, in this case, would refer to "component testing", which may be -confusing at times, especially when the concept of unit testing is again applied -to testing functions such as Redux `reducers` and `action creators` or any other -JavaScript function in your application. - -The other challenge that I often faced was whether to write tests for each -component in isolation or write a test for a feature that encapsulates a set of -related components. The later would be equivalent to writing what I would call -"integration tests". - -Finally, one would say "well then you could have just written the tests in a way -that resemble the way the application is used"! This approach is commonly -recommended in the React community, but it quickly becomes really complex to -maintain the layers of separation between **_unit tests_**, **_integration -tests_** and **_end-to-end tests_**. - -## What did I learn? - -Given a React/Redux application, here is how I would organise my testing -strategy: - -### Unit Tests - -- In a React app, **unit tests** will largely apply to testing "helper - functions" and not to testing components, as justified in the next section. - Helper functions, in this case, would refer to functions that live outside the - components and are neither Redux action creators nor reducers. These functions - can be used inside components, action creators, reducers or other parts of - your application. - -- Writing unit tests for "helper functions" would ensure their signatures and - expected outputs are protected against regressions. This would also ensure - their use across components or other functions is consistent and as expected. - -- Where possible, each "helper function" must have its own `unit test`. - -- An example of a unit test would like: - - ```javascript - const sum = require('../../js/sum'); - - test('adds 1 + 2 to equal 3', () => { - expect(sum(1, 2)).toBe(3); - }); - ``` - -- Write a _thousand_ of these. - -### Integration Tests - -- In the context of a React/Redux app, component tests can be thought of as - **integration tests**. This is because React components are built around - features such as ``, ``, etc. So one React component can - be a mix of other components to achieve a UI feature set. - -- To test a component, write an **integration test** that covers the use of a - given component for a given UI feature. - -- If a component being tested dispatches a Redux `action`, this is the right - place to test those actions and their effect on the UI. - -- Pay attention to the concept of _feature isolation_ vs _component isolation_ - as it will help you write better integration tests and also easily mock - component contexts. - -- A classic example of **feature isolation** is when you have a `` - component which displays a list of users and has a `