Interface - DO NOT MERGE - Merge bg:interface instead. ARCHIVED FOR CONVERSATION

[zthatch]

Basically looks and works just like PDFitc but on top of FastAPI. I also am yet to accept user data into a database or to return results as a zip file. Another difference is that there is also access to the rest api documentation for easy integration with other applications.

Next big steps are likely to change the testing such that it uses a fake google cloud store (maybe add some tests for the route calls), and then to (put in a strict rate limiter https://slowapi.readthedocs.io/en/latest/ and...) containerize it and try throwing it at cloud-run or appengine so that a select few can open it in their browser.

We likely also want to consider not exporting the data to google cloud storage. My understanding is that the IUCR maintains their mongo instance manually (rather than through atlas) and therefore aren't paying premiums for storage space. Probably good to keep both options open, but I should reiterate that keeping the data in the mongo database allows for aggregations, which are effectively distributed computing.

[zthatch]

[zthatch]

[sbillinge]

This looks great! Pretty exciting. Do we have a good definition of what the deliverable is before we get too far beyond a prototype/pilot? This should definitely be discussed. S

…

On Mon, Aug 30, 2021 at 9:49 PM zthatch ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/40968440/131428547-16cb23fe-de65-4484-94cc-859dec9d35a8.png> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAOWUI6BCHAZNXFABWCZTLT7QYL7ANCNFSM5DDBNOPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Simon Billinge Professor, Columbia University Physicist, Brookhaven National Laboratory

[zthatch]

This looks great! Pretty exciting. Do we have a good definition of what the deliverable is before we get too far beyond a prototype/pilot? This should definitely be discussed. S
…

My understanding from the last time that we met is that my part of the deliverable is to get an interface on the web similar to what the CLI tool is able to produce (achieved here, but not deployed), and to connect it to group resources on the backend (group managed atlas and gcp), which has not yet been achieved since I have been planning on grouping that development process with the deployment development process.

Berrak and Martin are working in parallel to increase the # of cifs that can be inserted into the database.

[sbillinge]

OK, we should talk about scope. This seems more than I had in mind initially. That's not to say we don't want all that, but I don't think it is clear in my mind where we want to head with this in general beyond the working demo we already have, and possibly taking it further to see how well the concept scales as # cifs goes up. Our public facing cloud product is PDFitc. This will be deployed by IUCr if they want and they should handle the database infrastructure. One thought I have is to include this as an App on PDFitc (why not? upload a pattern, get a list of relevant papers, so it looks a lot like all our other Apps so far). Another thought is to re-engineer the PDFitc backend in this way, so the work you have done to make fastAPI look like PDFitc is certainly a propos. But that decision hasn't been made. Do we know how stable and well supported the fastAPI project is that we would want to adopt it in production?

…

On Tue, Aug 31, 2021 at 1:51 PM zthatch ***@***.***> wrote: This looks great! Pretty exciting. Do we have a good definition of what the deliverable is before we get too far beyond a prototype/pilot? This should definitely be discussed. S … <#m_6117038161826083596_> On Mon, Aug 30, 2021 at 9:49 PM zthatch *@*.***> wrote: [image: image] https://user-images.githubusercontent.com/40968440/131428547-16cb23fe-de65-4484-94cc-859dec9d35a8.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#78 (comment) <#78 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABAOWUI6BCHAZNXFABWCZTLT7QYL7ANCNFSM5DDBNOPA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub . -- Simon Billinge Professor, Columbia University Physicist, Brookhaven National Laboratory My understanding from the last time that we met is that my part of the deliverable is to get an interface on the web similar to what the CLI tool is able to produce (achieved here, but not deployed), and to connect it to group resources on the backend (group managed atlas and gcp), which has not yet been achieved since I have been planning on grouping that development process with the deployment development process. Berrak and Martin are working in parallel to increase the # of cifs that can be inserted into the database. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABAOWUIOZFPDOIMGFDLJPEDT7UJB3ANCNFSM5DDBNOPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Simon Billinge Professor, Columbia University Physicist, Brookhaven National Laboratory

[zthatch]

OK, we should talk about scope. This seems more than I had in mind initially. That's not to say we don't want all that, but I don't think it is clear in my mind where we want to head with this in general beyond the working demo we already have, and possibly taking it further to see how well the concept scales as # cifs goes up. Our public facing cloud product is PDFitc. This will be deployed by IUCr if they want and they should handle the database infrastructure. One thought I have is to include this as an App on PDFitc (why not? upload a pattern, get a list of relevant papers, so it looks a lot like all our other Apps so far). Another thought is to re-engineer the PDFitc backend in this way, so the work you have done to make fastAPI look like PDFitc is certainly a propos. But that decision hasn't been made. Do we know how stable and well supported the fastAPI project is that we would want to adopt it in production?

Understood. My intention here was simply to show an exemplary web interface that utilized the plotting and printing code that has already been developed, using html and css that had already been written. (in such a way that it can be safely deployed as an example (left panel app) and a reference for front-end developers (right panel app))

With regard to future scope.

I guess the question is, are we giving IUCr a restAPI application with a codebase for them to deploy at their leisure, or are we giving them access to a restAPI that we deploy. I thought the former.

I don't think pdfitc should switch to fastapi, even though their base APIs (function signatures and names) are very similar. Fastapi is super popular right now, especially for developing a fast restAPI, like what I've done here, but flask has so many plug-ins (e.g. flask-auth, flask-...) that have their own API are being utilized on pdfitc that would likely be difficult to transition. You can also get most of the benefits of fast-api with a little refactoring. Also fastAPI is 2 years old and Flask is 12 or so. (that said, fast-API is so good that a lot of people are willing to make the switch, and have)

Flask supports the creation of RestAPIs, which I am very interested in adding to pdfitc, but I didn't think this was going to become part of that. The good news is that this mostly uses the base-level framework of fastapi, which is very similar to flask.