{"id":478,"date":"2016-11-03T18:38:37","date_gmt":"2016-11-03T18:38:37","guid":{"rendered":"http:\/\/marcedit.reeset.net\/learning_marcedit\/?page_id=478"},"modified":"2016-11-03T18:44:10","modified_gmt":"2016-11-03T18:44:10","slug":"chapter-5-batch-processing-records-02","status":"publish","type":"page","link":"https:\/\/marcedit.reeset.net\/learning_marcedit\/9-2\/chapter-5-batch-processing-records-02\/","title":{"rendered":"Chapter 5: Batch Processing Records"},"content":{"rendered":"<h2>In this Chapter:<\/h2>\n<ul>\n<li><strong>Getting Started<\/strong><\/li>\n<li><strong>Use Cases<\/strong><\/li>\n<li><strong>Batch Processing Data<\/strong><\/li>\n<\/ul>\n<h3>Getting Started<\/h3>\n<p>Libraries, today, receive metadata from a number of different sources and in a multitude of formats. \u00a0While many vendors still provide metadata as a single, large MARC download, other vendors may provide data in MARCXML, ONIX, or a myriad of other metadata formats. \u00a0In these cases, libraries will likely have to transform these non-MARC metadata formats into MARC if they wish to load the data into their ILS systems. \u00a0For the odd metadata file, pre-processing one file outside of the normal MARC workflow isn&#8217;t that big of a deal. \u00a0However, if your organization receives 100, or 1000, or in the case of one particular vendor that I&#8217;ve come across, 17,000 individual metadata files &#8212; the ability to touch and process these records becomes excessive. \u00a0This is what the batch processing tooling was designed to address &#8212; those messy cases where libraries have lots of individual files or metadata records that need to be processed before something else can be done to them.<\/p>\n<h3>Use Cases<\/h3>\n<p>As noted above, the batch processing tooling was designed to address a very specific set of use cases. \u00a0While the questions and data formats of the requests very, the primary two use cases can be summed up below:<\/p>\n<p><em>My vendor has provided me 17,000 individual MARCXML files. \u00a0I need to turn these into MARC for processing and loading into my ILS system.<\/em><\/p>\n<p><em>I&#8217;ve exported a set of records from my ILS and split them into separate files. \u00a0Is there a way that I can turn these into MARCXML or Dublin Core records without touching each file.<\/em><\/p>\n<p>Because MarcEdit provides tooling that allows users to join and split records once they are in the MARC format, the batch processing tool has been created to provide an automated method to perform large scale format transformations data in folders and at the folder\/subfolder level.<\/p>\n<h3>Batch Processing Records<\/h3>\n<p>The batch processing tooling\u00a0can be accessed from the Tool menu from the Main Application Window. \u00a0This tool can also be added to the Main Application Window by selecting the tool on the Preferences window.<\/p>\n<div id=\"attachment_484\" style=\"width: 815px\" class=\"wp-caption alignnone\"><a href=\"http:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1.png\"><img aria-describedby=\"caption-attachment-484\" decoding=\"async\" loading=\"lazy\" class=\"wp-image-484 size-full\" src=\"http:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1.png\" alt=\"batch_processing_window1\" width=\"805\" height=\"483\" srcset=\"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1.png 805w, https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1-300x180.png 300w, https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1-768x461.png 768w, https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-content\/uploads\/2016\/11\/batch_processing_window1-624x374.png 624w\" sizes=\"(max-width: 805px) 100vw, 805px\" \/><\/a><p id=\"caption-attachment-484\" class=\"wp-caption-text\">Figure 1: Batch Processing Dialog<\/p><\/div>\n<h4><strong>Processing Records<\/strong><\/h4>\n<p>When utilizing the Batch Process Tool, the application exposes all the available data transformation functions registered within the application. \u00a0This means that all the data processing tools available within the MARC Tools window are available as part of the Batch Processing Tool.<\/p>\n<p>The tool requires the user to provide a few pieces of information in order to perform the requested operations. \u00a0This information is as follows:<\/p>\n<p><strong>Source Directory:\u00a0<\/strong> The Source Directory represents\u00a0the top level directory with the files that will be processed. \u00a0For example, if a user had 3 records in the following directory structure:<\/p>\n<p>&nbsp;<\/p>\n<p>The source directory would be the full path to folder 1. \u00a0The same would be true if this example had included subfolders. \u00a0For example:<\/p>\n<pre>- folder 1 \r\n   - file 1 \r\n   - file 2 \r\n   - file 3\r\n<\/pre>\n<p><!--nextpage--><\/p>\n<pre>- folder 1 \r\n  - file 1 \r\n  - file 2 \r\n  - subfolder 1 \r\n    - file 1 \r\n    - file 2 \r\n  - subfolder 2 \r\n    - file 1 \r\n    - file 2 \r\n    - subsubfolder 3 \r\n      - file 1\r\n<\/pre>\n<p>The user would set the source directory at the level within the folder tree where file processing should begin. \u00a0If the user wanted to process all files (regardless of subfolder), the source directory would be set to folder 1. \u00a0If the user only wanted to process the files in subfolder 2, the source directory would be set to subfolder 2 and the option to process subfolders would be left unchecked.<\/p>\n<p><strong>Include Subfolders<\/strong>: This option let&#8217;s MarcEdit know if it should process just the records in the specified source directory, or if it should recursively process subfolders and apply the same processing instructions to files found within the subfolders under the source directory.<\/p>\n<p><strong>File Types: \u00a0<\/strong>This is a comma delimited list of file extensions that should be evaluated by the tool. \u00a0If, for example, the process was evaluating MARCXML data &#8212; the file type would be .xml.<\/p>\n<p><strong>Function: \u00a0<\/strong>All data transformation functions registered within MarcEdit can be accessed from this list. \u00a0Users just need to locate the process that they wish to run, and select it.<\/p>\n<p><strong>Step-by-Step<\/strong><\/p>\n<p>So, that was a lot of text. \u00a0In a nutshell, what are the steps to making this work?<\/p>\n<ol>\n<li>Select your Source Directory\u00a0(this is the folder with the data to be processed)<\/li>\n<li>Decide if you need to process subfolders as part of this process. \u00a0If you do, check this option.<\/li>\n<li>Select your file types to be processed by entering their file extension.<\/li>\n<li>Select the data transformation function from the list of functions.<\/li>\n<li>Click Next<\/li>\n<li>Wait for the results<\/li>\n<\/ol>\n<p>And that\u2019s it. \u00a0The Batch Processing\u00a0tool will provide a running status message so users can monitor the process and will report any files that were unable to be handled by the selected transformation.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In this Chapter: Getting Started Use Cases Batch Processing Data Getting Started Libraries, today, receive metadata from a number of different sources and in a multitude of formats. \u00a0While many vendors still provide metadata as a single, large MARC download, other vendors may provide data in MARCXML, ONIX, or a myriad of other metadata formats. [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":9,"menu_order":5,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"_links":{"self":[{"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/pages\/478"}],"collection":[{"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/comments?post=478"}],"version-history":[{"count":4,"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/pages\/478\/revisions"}],"predecessor-version":[{"id":489,"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/pages\/478\/revisions\/489"}],"up":[{"embeddable":true,"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/pages\/9"}],"wp:attachment":[{"href":"https:\/\/marcedit.reeset.net\/learning_marcedit\/wp-json\/wp\/v2\/media?parent=478"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}