Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Story/cite 163 - - We need a bulk upload for files. #23

Closed
wants to merge 47 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
7b2b4a9
[CITE-163] Add Giles Implementation
Charishma0249 Nov 23, 2021
ef6d573
[CITE-163] Giles Implementation
Charishma0249 Jun 9, 2022
afea37f
[CITE-163] Giles Implementation
Charishma0249 Jun 10, 2022
179f0b5
[CITE-163] Adding OAuth properties
Charishma0249 Jun 10, 2022
3a562d5
[CITE-163] Added collection import kafka topic
PradnyaC11 Aug 27, 2024
6a61a1f
[CITE-163] Added supporting methods to process the file
PradnyaC11 Aug 29, 2024
45f6008
[CITE-163] Added code to process the collection import
PradnyaC11 Aug 30, 2024
73f3db3
[CITE-163] Removed unwanted old code
PradnyaC11 Sep 27, 2024
d97177e
[CITE-163] Added BibFileHandler
PradnyaC11 Oct 4, 2024
6235fec
[CITE-163] Added Bib file iterator code
PradnyaC11 Oct 8, 2024
264a5ba
[CITE-163] Updated BibFile Iterator
PradnyaC11 Oct 9, 2024
c05de5c
[CITE-163] Parsed article Type of bib file
PradnyaC11 Oct 10, 2024
f3cee64
[CITE-163] Parsed data for Bib Iterator
PradnyaC11 Oct 11, 2024
46fc54a
[CITE-163] Added code to parse author list
PradnyaC11 Oct 15, 2024
318f193
[CITE-163] Parsed some more details of citation
PradnyaC11 Oct 16, 2024
d281d39
[CITE-163] Created item from bib file text
PradnyaC11 Oct 17, 2024
91925ec
[CITE-163] Parsed Citation to BibEntry in BibFileIterator
PradnyaC11 Oct 18, 2024
3d2802f
[CITE-163] Mapped references and authors from citation to bibentry
PradnyaC11 Oct 22, 2024
3717bf4
[CITE-163] Solved some errors that bib Iterator
PradnyaC11 Oct 23, 2024
2fb3a4f
[CITE-163] Changed the logic to parse extra/note
PradnyaC11 Oct 24, 2024
1477e37
[CITE-163] Cleaned code and resolved error related to parsing
PradnyaC11 Oct 25, 2024
86078c5
[CITE-163] Resolving parsing note issues
PradnyaC11 Nov 1, 2024
d6305ef
[CITE-163] Resolving error with JSON parser
PradnyaC11 Nov 5, 2024
304cb17
[CITE-163] Correctly parsed extra and note
PradnyaC11 Nov 6, 2024
74c0248
[CITE-163] Resolved authors & editors issue
PradnyaC11 Nov 7, 2024
3f2f3fc
[CITE-163] Resolved DOI & ISBN parsing issue
PradnyaC11 Nov 8, 2024
2b3a888
[CITE-163] Added collectionId to JobInfo, so as to import to it
PradnyaC11 Nov 15, 2024
fb3fb21
[CITE-163] Cleaned code for bibFileIterator
PradnyaC11 Nov 18, 2024
35075f9
[CITE-163] Resolving issue with CollectionIds
PradnyaC11 Nov 19, 2024
604a621
[CITE-163] Resolved importing to collection issue
PradnyaC11 Nov 20, 2024
e9cbdbe
[CITE-163] Added file path to Article Meta
PradnyaC11 Nov 27, 2024
bb79bdd
[CITE-163] Added method to CitesphereConnector to getItem
PradnyaC11 Dec 2, 2024
b51c18c
[CITE-163] Added uploadFile & makePostApiCall to CitesphereConnector
PradnyaC11 Jan 24, 2025
ab8fb23
[CITE-163] Added Giles connector & supporting classes
PradnyaC11 Jan 28, 2025
0b3af9b
[CITE-163] Updated GilesConnector with Giles token from job info
PradnyaC11 Jan 30, 2025
9081531
[CITE-163] Cleaned-up code & fixed some errors
PradnyaC11 Jan 31, 2025
98319b6
[CITE-163] Updated giles upload part to map files with items
PradnyaC11 Feb 4, 2025
26bc2b7
[CITE-163] Updated Citesphere cinnecter get Item mapping
PradnyaC11 Feb 6, 2025
91c1380
[CITE-163] Updated BibEntry and ExtraFieldHelper for giles upload
PradnyaC11 Feb 7, 2025
7d4c332
[CITE-163] Updated bibFileIterator & tested the giles upload
PradnyaC11 Feb 11, 2025
f732652
[CITE-163] Address the User class error
PradnyaC11 Feb 18, 2025
fd79011
[CITE-163] Fixed user issue in giles connector
PradnyaC11 Feb 20, 2025
b890552
[CITE-163] Resolved the null pointer exception related to GilesConnector
PradnyaC11 Feb 21, 2025
725c08b
[CITEE-163] Cleaned up code
PradnyaC11 Feb 27, 2025
7c5d730
[CITE-163] Removed unwanted changes
PradnyaC11 Feb 27, 2025
4aac546
[CITE-163] Resolving code factor issues
PradnyaC11 Feb 28, 2025
8d64005
[CITE-163] Resolving code factor issues
PradnyaC11 Feb 28, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions citesphere-importer/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@
<spring-data.version>Lovelace-RELEASE</spring-data.version>
<thymeleaf.version>3.0.11.RELEASE</thymeleaf.version>
<spring.kafka.version>2.2.6.RELEASE</spring.kafka.version>
<citesphere.messages.version>0.2</citesphere.messages.version>
<citesphere.messages.version>0.6</citesphere.messages.version>
<citesphere.model.version>1.24</citesphere.model.version>
<spring-security-oauth2-version>2.4.0.RELEASE</spring-security-oauth2-version>

<admin.password>$2a$04$oQo44vqcDIFRoYKiAXoNheurzkwX9dcNmowvTX/hsWuBMwijqn44i</admin.password>

Expand Down Expand Up @@ -69,7 +71,11 @@
<artifactId>citesphere-messages</artifactId>
<version>${citesphere.messages.version}</version>
</dependency>

<dependency>
<groupId>edu.asu.diging</groupId>
<artifactId>citesphere-model</artifactId>
<version>${citesphere.model.version}</version>
</dependency>
<!-- Spring -->
<dependency>
<groupId>org.springframework</groupId>
Expand Down Expand Up @@ -122,6 +128,12 @@
<artifactId>spring-aop</artifactId>
</dependency>

<dependency>
<groupId>org.springframework.security.oauth</groupId>
<artifactId>spring-security-oauth2</artifactId>
<version>${spring-security-oauth2-version}</version>
</dependency>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this needed?


<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
Expand Down Expand Up @@ -302,6 +314,12 @@
<version>1.10.19</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<scope>test</scope>
</dependency>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is that used?


<dependency>
<groupId>com.opencsv</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.kafka.annotation.KafkaListener;

import com.fasterxml.jackson.databind.ObjectMapper;
Expand All @@ -18,7 +19,12 @@ public class ReferenceImportListener {
private final Logger logger = LoggerFactory.getLogger(getClass());

@Autowired
private IImportProcessor processor;
@Qualifier("importProcessor")
private IImportProcessor fileProcessor;

@Autowired
@Qualifier("collectionImportProcessor")
private IImportProcessor collectionProcessor;

@KafkaListener(topics = KafkaTopics.REFERENCES_IMPORT_TOPIC)
public void receiveMessage(String message) {
Expand All @@ -32,6 +38,21 @@ public void receiveMessage(String message) {
return;
}

processor.process(msg);
fileProcessor.process(msg);
}

@KafkaListener(topics = KafkaTopics.COLLECTION_IMPORT_TOPIC)
public void receiveCollectionImportMessage(String message) {
ObjectMapper mapper = new ObjectMapper();
KafkaJobMessage msg = null;
try {
msg = mapper.readValue(message, KafkaJobMessage.class);
} catch (IOException e) {
logger.error("Could not unmarshall message.", e);
return;
}

collectionProcessor.process(msg);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
public class Affiliation {

private String name;
private String uri;
private String localAuthorityId;

public String getName() {
return name;
Expand All @@ -11,4 +13,22 @@ public String getName() {
public void setName(String name) {
this.name = name;
}

public String getUri() {
return uri;
}

public void setUri(String uri) {
this.uri = uri;
}

public String getLocalAuthorityId() {
return localAuthorityId;
}

public void setLocalAuthorityId(String localAuthorityId) {
this.localAuthorityId = localAuthorityId;
}


}
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,16 @@
import java.util.ArrayList;
import java.util.List;

import com.fasterxml.jackson.annotation.JsonProperty;

import edu.asu.diging.citesphere.model.bib.IGilesUpload;

public class ArticleMeta {

private List<String> collections;
private List<ArticleId> articleIds;
private String articleTitle;
private String articleShortTitle;
private List<ArticleCategoryGroup> categoryGroups = new ArrayList<ArticleCategoryGroup>();
private List<Contributor> contributors;
private String authorNotesCorrespondence;
Expand All @@ -25,10 +31,13 @@ public class ArticleMeta {
private String copyrightYear;
private String copyrightHolder;
private String selfUri;
@JsonProperty("DOI")
private String doi;
private String articleAbstract;
private String language;
private ReviewInfo reviewInfo;
private String documentType;
private String filePath;
private String conferenceTitle;
private String conferenceDate;
private String conferenceLocation;
Expand All @@ -43,7 +52,14 @@ public class ArticleMeta {
private List<Reference> references;
private String referenceCount;
private String retrievalDate;
private List<IGilesUpload> gilesUpload;

public List<String> getCollections() {
return collections;
}
public void setCollections(List<String> collections) {
this.collections = collections;
}
public List<ArticleId> getArticleIds() {
return articleIds;
}
Expand All @@ -56,6 +72,12 @@ public String getArticleTitle() {
public void setArticleTitle(String articleTitle) {
this.articleTitle = articleTitle;
}
public String getArticleShortTitle() {
return articleShortTitle;
}
public void setArticleShortTitle(String articleShortTitle) {
this.articleShortTitle = articleShortTitle;
}
public List<ArticleCategoryGroup> getCategories() {
return categoryGroups;
}
Expand Down Expand Up @@ -164,6 +186,12 @@ public String getSelfUri() {
public void setSelfUri(String selfUri) {
this.selfUri = selfUri;
}
public String getDoi() {
return doi;
}
public void setDoi(String doi) {
this.doi = doi;
}
public String getArticleAbstract() {
return articleAbstract;
}
Expand All @@ -188,6 +216,12 @@ public String getDocumentType() {
public void setDocumentType(String documentType) {
this.documentType = documentType;
}
public String getFilePath() {
return filePath;
}
public void setFilePath(String filePath) {
this.filePath = filePath;
}
public String getConferenceTitle() {
return conferenceTitle;
}
Expand Down Expand Up @@ -272,5 +306,11 @@ public String getRetrievalDate() {
public void setRetrievalDate(String retrievalDate) {
this.retrievalDate = retrievalDate;
}
public List<IGilesUpload> getGilesUpload() {
return gilesUpload;
}
public void setGilesUpload(List<IGilesUpload> gilesUpload) {
this.gilesUpload = gilesUpload;
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ public class Contributor {
private String fullSurname;
private String fullName;
private String emailAddress;
private String uri;
private List<Affiliation> affiliations = new ArrayList<Affiliation>();
private List<ContributorId> ids;

Expand Down Expand Up @@ -64,6 +65,15 @@ public String getEmailAddress() {
public void setEmailAddress(String emailAddress) {
this.emailAddress = emailAddress;
}
public String getUri() {
return uri;
}
public void setUri(String uri) {
this.uri = uri;
}
public void setFullStandardizeName(String fullStandardizeName) {
this.fullStandardizeName = fullStandardizeName;
}
public List<Affiliation> getAffiliations() {
return affiliations;
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package edu.asu.diging.citesphere.importer.core.service;

import edu.asu.diging.citesphere.model.bib.IGilesUpload;

public interface IGilesConnector {

IGilesUpload uploadFile(String username, String token, String filename, byte[] fileBytes);
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
package edu.asu.diging.citesphere.importer.core.service.giles.impl;

import javax.annotation.PostConstruct;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.PropertySource;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Service;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.RestTemplate;

import edu.asu.diging.citesphere.importer.core.service.IGilesConnector;
import edu.asu.diging.citesphere.model.bib.IGilesUpload;
import edu.asu.diging.citesphere.model.bib.impl.GilesUpload;

@Service
@PropertySource("classpath:/config.properties")
public class GilesConnector implements IGilesConnector {

private final Logger logger = LoggerFactory.getLogger(getClass());

private RestTemplate restTemplate;

@Value("${_giles_baseurl}")
private String gilesBaseurl;

@Value("${_giles_upload_endpoint}")
private String uploadEndpoint;

@PostConstruct
public void init() {
restTemplate = new RestTemplate();
}

@Override
public IGilesUpload uploadFile(String username, String token, String filename, byte[] fileBytes) {

if (restTemplate == null) {
restTemplate = new RestTemplate(); // Ensure restTemplate is initialized
}

HttpHeaders headers = new HttpHeaders();
headers.setContentType(MediaType.MULTIPART_FORM_DATA);
headers.setBearerAuth(token);

MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("files", new MultipartFileResource(fileBytes, filename));

HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);

ResponseEntity<UploadResponse> response = null;
try {
response = restTemplate.exchange(gilesBaseurl + uploadEndpoint, HttpMethod.POST, requestEntity, UploadResponse.class);
} catch (Exception ex) {
logger.error("Upload request failed", ex);
return null;
}
IGilesUpload upload = new GilesUpload();
upload.setProgressId(response.getBody().getId());
upload.setUploadingUser(username);
return upload;
}

public class MultipartFileResource extends ByteArrayResource {

private String filename;

public MultipartFileResource(byte[] bytearray, String filename) {
super(bytearray);
this.filename = filename;
}

@Override
public String getFilename() {
return this.filename;
}
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
package edu.asu.diging.citesphere.importer.core.service.giles.impl;

class UploadResponse {
private String id;
private String checkUrl;

public String getId() {
return id;
}

public void setId(String id) {
this.id = id;
}

public String getCheckUrl() {
return checkUrl;
}

public void setCheckUrl(String checkUrl) {
this.checkUrl = checkUrl;
}
}
Loading