Powered by CERN Data Centre & InvenioRDM

Powered by CERN Data Centre & InvenioRDM

Powered by CERN Data Centre & InvenioRDM

Powered by CERN Data Centre & InvenioRDM

Starting this week blog posts are archived in the Rogue Scholar science blog archive using a dedicated invenio module instead of an external service. This simplifies the maintenance of the service and is critical for the long-term future of the archived content. Rogue Scholar started out as a bespoke service written in Javascript in early 2023. In September 2024 Rogue Scholar started the migration to the InvenioRDM repository platform. This migration is now complete with the release of the invenio-feeds module that provides functionality to automatically parse blog feeds and archive the metadata and content in a digital repository. This extends functionality already available in InvenioRDM, or added in previous work – DOI registration with Crossref, full-text search, and automated subject classification with the OpenAlex vocabulary falls in the later category. Until the release of invenio-feeds, automatic parsing of newly published or updated science blog posts was done by a dedicated Javascript and later Python service hosted at api.rogue-scholar.org. This service is no longer needed and will be retired on November 1st, reducing the complexity and cost of Rogue Scholar infrastructure. One example is the use of GitHub Actions to trigger the parsing of all blog feeds every 10 min which is now handled by InvenioRDM background workers. The switch from dedicated external service to Python module has been fairly smooth so far, but I absolutely expect issues to come up in the coming weeks and months. The migration to invenio module not only simplifies Rogue Scholar infrastructure, but comes with new functionality: archiving of blog posts as PDF files in the PDF/A format as InvenioRDM file attachments. One fundamental archiving principle is LOCKSS (Lots of Copies Keep Stuff Safe), and these PDF files nicely complement archiving activities by the blog itself, archiving of metadata and full-text in Rogue Scholar, and archiving of participating science blogs in the Internet Archive Archive-It service. These PDF files are generated the moment content is added or updated to Rogue Scholar, and it will take a few months to address issues with the PDF files (e.g. pagination, image sizing), and archiving the more than 50K science blog posts in Rogue Scholar. The PDF files use the same layout as the PDF files generated by Rogue Scholar since January 2024, but are generated with every submission to Rogue Scholar instead of dynamically on demand (using the commonmeta-py library). The PDF files are again generated with the WeasyPrint and pikepdf Python libraries, but no longer use Pandoc and the markdown format as an intermediary step. Other formats (markdown, ePub or JATS XML) are no longer supported, and this simplification makes the PDF generation workflow simpler and faster – allowing me to generate more than 3000 PDF attachments this week. I have a painful relationship with PDF and scholarly publishing, and have long thought the proper archiving format for Rogue Scholar. WARC is a widely used archiving format developed and used by the Internet Archive – supported in InvenioRDM since the recent v14 release. Markdown and ePub are open formats closer to the technologies used by blogging platforms. PDF is fine for reading and archiving, but very painful to work with to extract content out again. The decision for PDF/A (and not WARC or ePub) was mainly for two reasons: * a widely supported official archiving format (e.g. digital repositories), * widely used by scholars and supported by the tools and workflow they use. The PDF/A that Rogue Scholar generates has two important features that overcome critical PDF shortcomings: * rich metadata in XMP format, including DOI, authors, title, abstract and license * full-text content in HTML format as attachment that can easily be accessed, using the PDF/A-3a standard. Over the coming months I will not only work on any issues that come up with the new invenio-feeds module and PDF generation, but reach out to participating science blogs about archiving the PDF/A files in associated repositories, starting with the Invenio blog and Zenodo. Please reach out via Slack, email, Mastodon, or Bluesky if you have any questions or comments. Rogue Scholar is a scholarly infrastructure that is free for all authors and readers. You can support Rogue Scholar with a one-time or recurring donation or by becoming a sponsor. ## References 1. Fenner, M. (2022, December 12). Building an archive for scholarly blog posts. _Front Matter_. https://doi.org/10.53731/br9f5xa-a556w2t 2. Fenner, M. (2024, September 2). Rogue Scholar migrates to InvenioRDM. _Front Matter_. https://doi.org/10.53731/sdazp-kzn55 3. Fenner, M. (2023, October 30). Starting November, all Rogue Scholar blog posts will be archived by the Internet Archive. _Front Matter_. https://doi.org/10.53731/hhtx0-wb293 4. Fenner, M. (2024, January 8). Every Rogue Scholar blog post now available in Markdown, ePub, and PDF formats. _Front Matter_. https://doi.org/10.53731/1dfxr-hs665 5. Fenner, M. (2010, October 6). Beyond the PDF – it is time for a workshop. _Front Matter_. https://doi.org/10.53731/r294649-6f79289-8cw7z

Starting this week the Rogue Scholar science blog archive collects citations of its blog posts using invenio-pidbox, the Python module for the InvenioRDM repository platform that I launched earlier this month, and replacing the integration that used the Crossref cited-by service launched in August 2025. The reasons for this change are threefold: * a simplified architecture that no longer needs a dedicated citations database table and service external to the InvenioRDM repository platform, * being able to track citations of blog posts registered with DataCite, and citations by DataCite DOIs, * by not using an external servive there are no API rate-limits to observe. invenio-pidbox uses a sqlite database to store scholarly metadata and uses the commonmeta-py Python library and its Rust bindings to query sqlite and integrate with the InvenioRDM repository platform that Rogue Scholar uses. The initial invenio-pidbox versions used the ROR and ORCID regular data dumps (135,710 organizations and 25,048,058 people, respectively) to generate author and organization profiles. With this release I started to also store Crossref and DataCite metadata in sqlite. Initially only the metadata for the blog posts in Rogue Scholar, so only tracking citations by other blog posts archived in Rogue Scholar, but the architecture allows me to scale to all 180 million Crossref DOIs and 120 million DataCite DOIs in their annual data dumps, and the about 2.4 billion references they contain, all in all about 2 TB of data. This sounds like a lot of data, but sqlite can easily handle this, and this is much easier and cheaper than importing all the metadata into Postgres and Opensearch, the database and search index used by InvenioRDM. invenio-pidbox comes with a celery background worker task to extracts all Rogue Scholar citations out of sqlite and stores them as InvenioRDM metadata so that they can be shown and queried in Rogue Scholar. Blog post citations can be much faster than citations by journal articles, e.g. five days after the publication of the InvenioRDM v14 announcement: Citations counts in Rogue Scholar are displayed for blogs, other communities, authors and institutions: Clicking on the citation count (here 58) opens a query for all cited blog posts. The DataCite blog is also an example for tracking citations of blog posts with DataCite DOIs, something that Rogue Scholar couldn't do until this week. invenio-pidbox is an invenio module that should work with all InvenioRDM v14 instances, and besides Rogue Scholar they currently all use DataCite DOIs for their content, making this an attractive option for other InvenioRDM instances that have 2 TB disk space to spare – not S3 object storage as sqlite needs fast disk access. Institution profiles are an interesting use case. Their main challenge is incomplete metadata, e.g only 137 of currently 683 posts in the Crossref blog have affiliation metadata. Affiliation metadata (using ROR) are more difficult than author metadata (using ORCID), as affiliations have a start and end date, and authors can have multiple affiliations in parallel. But with 300 million DOIs for works from Crossref and DataCite stored in sqlite in commonmeta format it becomes much easier to extract that information at scale, not limited by API rate-limits or differences between Crossref and DataCite metadata. Please reach out via Slack, email, Mastodon, or Bluesky if you have any questions or comments. Rogue Scholar is a scholarly infrastructure that is free for all authors and readers. You can support Rogue Scholar with a one-time or recurring donation or by becoming a sponsor. ## References 1. Fenner, M. (2026, August 12). InvenioRDM v14.0 and invenio-pidbox module released. _Front Matter_. https://doi.org/10.53731/fqvcp-0bw56 2. Fenner, M. (2025, August 4). Rogue Scholar citation tracking launches to production. _Front Matter_. https://doi.org/10.53731/zyg15-qv911 3. Viger, G. (2026, August 7). InvenioRDM v14.0 released. _Invenio_. https://doi.org/10.63517/kshzw-ay335

Last Friday the next major version (v14.0.0) of the InvenioRDM repository software was released after many months of work involving lots of people (listed in the blog post). The changes and upgrade instructions are documented in detail the post on the Invenio blog. The most important change for the Rogue Scholar science blog archive, and the feature that I contributed, is the support added for Crossref DOI registration. Rogue Scholar was running on v14.0.0 release candidate versions the past few months, so the upgrade to v14.0.0 was easy and happened on Friday. As part of the upgrade I changed the self-hosted web analytics service from Plausible Analytics to Hitkeep, mainly because its technology stack (Go, DuckDB) is a better fit for Rogue Scholar than Plausible’s (Elixir, Clickhouse, Postgres). Please reach out to me if you need help with cloud hosting and/or commercial support, e.g. because you want to start an InvenioRDM repository with Crossref DOI registration for preprints, dissertations or other textual content. This week I released invenio-pidbox, an invenio module for InvenioRDM v14 that integrates a read-only sqlite database for content that is too large for InvenioRDM vocabularies (typically yaml or json format), in my case the ORCID Public Data File 2025 with 25 million records. The ORCID data in compressed XML format are converted into commonmeta JSON according to the new Commonmeta v1.0 schema, and integrated into InvenioRDM using commonmeta-py and the commonmeta-py extra commonmeta-rs with Rust bindings and sqlite support. This allows both ORCID metadata lookup as well as ORCID queries from within InvenioRDM. For example in the records submission form: Or the Rogue Scholar author profile: Or the search for an author by name: As I said before, ORCID metadata are used not only for author profiles, but also to handle names with multiple given or family names, and author affiliations – which need the work publication date and affiliation start and end dates. Affiliations are an important part of author information, and storing Research Organization Registry (ROR) metadata also in sqlite makes for an easier integration. InvenioRDM already supports automatic updates of ROR organization vocabularies, but only uses limited information. Invenio-pidbox knows all relevant information from ROR metadata and can show institution profiles, together with all publications from authors affiliated with that organization: And allows search of an organization by name: Institution profiles show basically the same information as the ROR website, plus all publications (in the case of Rogue Scholar blog posts). The ROR API allows up to 2000 requests within a 5 min period, which is more than enough for most use cases. If you need to make more requests or want to ensure faster response times, ROR suggests to run the ROR API locally. With exactly the same data you can also run the ROR API via sqlite and the invenio-pidbox integration, and the same goes for the ORCID API (but requires more work as more data are frequently updated). Putting all this into an invenio module instead of custom code in Rogue Scholar makes the functionality described above available to all InvenioRDM v14 instances, and I have for example installed the module with InvenioRDM Starter (demo website, documentation). InvenioRDM Starter facilitates deployment and configuration of InvenioRDM, allowing you to run InvenioRDM on your local computer within 15 min. This is achieved by providing * a prebuilt Invenio-App-RDM Docker image. * a Docker Compose configuration file with sensible defaults and searchable documentation. Invenio-pidbox is used in production in Rogue Scholar but still early in development, in particular needs better documentation for configuring sqlite and importing annual data files. All ORCID and ROR records are a combined 90 GB, as I store the original metadata as well as the metadata converted into commonmeta format. Please reach out via Slack, email, Mastodon, or Bluesky if you have any questions or comments. Rogue Scholar is a scholarly infrastructure that is free for all authors and readers. You can support Rogue Scholar with a one-time or recurring donation or by becoming a sponsor. ## References 1. Viger, G. (2026, August 7). InvenioRDM v14.0 released. _Invenio_. https://doi.org/10.63517/kshzw-ay335 2. ORCID. (2025). _ORCID Public Data File 2025_ (p. 237733993308 Bytes) [Dataset]. ORCID. https://doi.org/10.23640/07243.30375589 3. Fenner, M. (2026, July 15). Announcing the commonmeta schema 1.0 release candidate. _Front Matter_. https://doi.org/10.53731/kdqkf-nf052 4. Fenner, M. (2026, August 5). Rogue Scholar improves author profiles. _Front Matter_. https://doi.org/10.53731/ahkac-qfx88 5. Research Organization Registry. (2026). _ROR Data_ (Version v2.11) [Dataset]. Zenodo. https://doi.org/10.5281/ZENODO.6347574 6. Fenner, M. (2024, June 17). Announcing InvenioRDM Starter Beta. _Front Matter_. https://doi.org/10.53731/jxecm-0me48

The InvenioRDM project partners met in Hamburg two weeks ago, and we discussed a wide range of topics over five days. InvenioRDM is the open source repository platform that also powers the Rogue Scholar science blog archive, and one priority for me is to keep the repository platform's maintenance simple

Powered by CERN Data Centre & InvenioRDM

Ten days ago, I reported on a new deployment strategy for the InvenioRDM repository software. Using the Kamal deployment tool, I deployed both a staging instance of the Rogue Scholar service and a demo instance of the InvenioRDM Starter package. Over the last few days I have updated both instances

InvenioRDM is the open source turn-key research data management platform, with detailed documentation available here. InvenioRDM Starter facilitates deployment and configuration of InvenioRDM, allowing you to run InvenioRDM on your local computer within 15 min. This is achieved by providing a) a prebuilt Invenio-App-RDM Docker image, and b) a Docker

Powered by CERN Data Centre & InvenioRDM