Repository navigation
SONARJAVA-7341 Simplify the parser interface - #6517
aurelien-coet-sonarsource merged 4 commits into
Conversation
| public interface JParser { | ||
|
|
||
| JavaTree.CompilationUnitTreeImpl parse(String version, String unitName, String source); | ||
| JavaVersion MAXIMUM_SUPPORTED_JAVA_VERSION = new JavaVersionImpl(JavaVersionImpl.MAX_SUPPORTED, true); | ||
|
|
||
| enum Mode { | ||
| BATCH, | ||
| FILE_BY_FILE; | ||
|
|
||
| static JavaTree.CompilationUnitTreeImpl parse(JParser parser, String version, String unitName, String source) { | ||
| return parser.parse(version, unitName, source); | ||
| public JParser create(JavaVersion javaVersion, List<File> classpath) { | ||
| return create(javaVersion, classpath, false); | ||
| } | ||
|
|
||
| public JParser create(JavaVersion javaVersion, List<File> classpath, boolean shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| return JDTParser.create(this, javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| } |
There was a problem hiding this comment.
⚠️ Cross-Repo: SE test helper JParserTestUtils.parse stops compiling on java-frontend bump
Fix here and in SonarSource/sonar-java-symbolic-execution
Breaks in: SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java - JParserTestUtils.parse
This PR deletes org.sonar.java.model.JParserConfig, which takes astParser() and MAXIMUM_SUPPORTED_JAVA_VERSION with it. It also removes the static JParser.parse(JParser, String, String, String) helper and the instance method parse(String, String, String). JParser is left with only the multi-file parse(Iterable<InputFile>, BooleanSupplier, AnalysisProgress, BiConsumer<InputFile, JParseResult>).
In SonarSource/sonar-java-symbolic-execution, JParserTestUtils.parse (java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java) still uses JParserConfig.MAXIMUM_SUPPORTED_JAVA_VERSION, JParserConfig.Mode.FILE_BY_FILE.create(...).astParser() and the static JParser.parse(parser, ...). When that repo upgrades its pinned java-frontend to a build that includes this change, the helper no longer compiles, and neither does any SE test that depends on it.
The removal is deliberate, so the SE repo is the side that has to move. Because only the InputFile-based API is left, its test module also needs a test-scoped sensor-test-fixtures dependency (the same one this PR adds to java-checks-common/pom.xml) or some other way to get an InputFile. The 8.47 notes in documentation.md don't name the removed JParserConfig#astParser(), the static JParser.parse(JParser, …), or the move of MAXIMUM_SUPPORTED_JAVA_VERSION.
Here: In the 8.47 Breaking notes, list the removed JParserConfig#astParser(), JParserConfig#parse(String, String) and static JParser.parse(JParser, String, String, String), and say that MAXIMUM_SUPPORTED_JAVA_VERSION now lives on JParser. Then tell sonar-java-symbolic-execution so it can migrate to the InputFile-based parse when it bumps java-frontend.
There: In JParserTestUtils.parse, remove the JParserConfig, astParser() and static JParser.parse usage. Wrap the source in a TestInputFileBuilder InputFile and call the multi-file parse on JParser.Mode.FILE_BY_FILE.create(version, classpath). Then add a test-scoped org.sonarsource.scanner.engine:sensor-test-fixtures dependency to the plugin's pom.
In SonarSource/sonar-java-symbolic-execution - java-symbolic-execution/java-symbolic-execution-plugin/src/test/java/org/sonar/java/se/utils/JParserTestUtils.java:
public static CompilationUnitTree parse(String unitName, String source, List<File> classpath) {
JavaVersion version = JParser.MAXIMUM_SUPPORTED_JAVA_VERSION;
InputFile inputFile = new TestInputFileBuilder("", unitName)
.setContents(source).setCharset(StandardCharsets.UTF_8).build();
List<JParseResult> results = new ArrayList<>();
JParser.Mode.FILE_BY_FILE.create(version, classpath)
.parse(List.of(inputFile), () -> false, new AnalysisProgress(1), (input, result) -> results.add(result));
try {
return results.getFirst().get();
} catch (RuntimeException e) {
throw e;
} catch (Exception e) {
throw new IllegalStateException("Unable to parse source", e);
}
}
Was this helpful? React with 👍 / 👎
| public static JParser create(JParser.Mode mode, JavaVersion javaVersion, List<File> classpath, boolean shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| if (shouldIgnoreUnnamedModuleForSplitPackage) { | ||
| LOG.info("The Java analyzer will ignore the unnamed module for split packages."); | ||
| } | ||
| return ((JavaTree) node).getChildren().iterator(); | ||
| return switch (mode) { | ||
| case BATCH -> new Batch(javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| case FILE_BY_FILE -> new FileByFile(javaVersion, classpath, shouldIgnoreUnnamedModuleForSplitPackage); | ||
| }; | ||
| } | ||
|
|
||
| @VisibleForTesting | ||
| static List<Token> lex(String version, String unitName, char[] sourceChars) { | ||
| List<Token> tokens = new ArrayList<>(); | ||
| Scanner scanner = new Scanner( | ||
| true, | ||
| false, | ||
| false, | ||
| CompilerOptions.versionToJdkLevel(version), | ||
| null, | ||
| null, | ||
| false | ||
| ); | ||
| scanner.fakeInModule = "module-info.java".equals(unitName); | ||
| scanner.setSource(sourceChars); | ||
| while (true) { | ||
| try { | ||
| TerminalToken tokenType = scanner.getNextToken(); | ||
| Token token = Token.fromCurrent(scanner, tokenType); | ||
| tokens.add(token); | ||
| if (tokenType == TerminalToken.TokenNameEOF) { | ||
| break; | ||
| } | ||
| } catch (InvalidInputException e) { | ||
| throw new IllegalStateException(e); | ||
| } | ||
| } | ||
| return tokens; | ||
| } | ||
|
|
||
| private CompilationUnit compilationUnit; | ||
|
|
||
| private TokenManager tokenManager; | ||
| private LineColumnConverter lineColumnConverter; | ||
|
|
||
| private JDTSema sema; | ||
|
|
||
| private final Deque<JLabelSymbol> labels = new LinkedList<>(); | ||
|
|
||
| private void declaration(@Nullable IBinding binding, Tree node) { | ||
| if (binding == null) { | ||
| return; | ||
| } | ||
| sema.declarations.put(binding, node); | ||
| static boolean shouldEnablePreviewFlag(JavaVersion currentVersion) { | ||
| return currentVersion.arePreviewFeaturesEnabled(); | ||
| } | ||
|
|
There was a problem hiding this comment.
⚠️ Cross-PR: JDK release-option support from #6459 is lost after JParserConfig removal
Breaks when this PR and #6459 both merge into the same line. Whichever lands second either fails to compile or silently drops the sonar.java.useJdkReleaseOption behaviour.
How
- #6459 adds a
shouldUseJdkReleaseOptionfield, a four-argumentMode.create(...)anduseReleaseApiOfConfiguredJdk(...)toJParserConfig(java-frontend/src/main/java/org/sonar/java/model/JParserConfig.java:141). It calls them fromJavaFrontend.scanBatchandJavaAstScanner.scan. - This PR deletes
JParserConfig. Parser construction moves toJParser.Mode.create(JavaVersion, List<File>, boolean)(java-frontend/src/main/java/org/sonar/java/model/JParser.java:36-42) andJDTParser.create. JDTParser.jdtAstParser()(java-frontend/src/main/java/org/sonar/java/jdt/JDTParser.java:84-112) builds the ECJ environment from the raw classpath and never setsSystemReleaseClasspath.OPTION_RELEASE. If #6459's call sites are only ported toJParser.Mode, the parser keeps using the running JDK's API instead of the target release's.
Fix
- This PR: If #6459 lands first, add the
shouldUseJdkReleaseOptionflag toJParser.Mode.create,JDTParser.createand theJDTParserconstructor. MoveuseReleaseApiOfConfiguredJdkintoJDTParser.jdtAstParser()and keep theJavaFrontend/JavaAstScannerwiring. - #6459: If this PR lands first, rebase onto
JParser.Mode/JDTParser. Put the release-option logic inJDTParser.jdtAstParser()instead ofJParserConfig.astParser(), and move theJParserConfigTestcases toJParserModeTest/JDTParserTest.
Compared 5006a8c with #6459 at 71f4165
Was this helpful? React with 👍 / 👎
leonardo-pilastri-sonarsource
left a comment
There was a problem hiding this comment.
Looks good to me.
For the runtime risk of breaking things:
- DBD only relies on removed classes through the StubsGenerator, which is not run at runtime, so analysis on my local SQ with the new sonar-java was functioning correctly.
- Security also is not be affected on production code
- Same for symbolic-execution
b69f143 to
a4fde5d
Compare
|
Code Review
|
| Auto-apply | Compact | Unblock |
|
|
|
Was this helpful? React with 👍 / 👎 | Gitar




Summary by Gitar
JParserConfigwith aJParserinterface andJParser.Modefactories inJParser.javaIterable<InputFile>and reports results throughJParseResultJDTParserConfigintoJDTParser(batch and file-by-file) andJDTAstConverterorg.sonar.java.jdtwithJDTprefixesTypeandSymbolvalues instead of exposing ECJIBindingtypesJSymbolMetadata.JNullabilityDatamoved to top-levelorg.sonar.java.model.JNullabilityDatadocumentation.mdThis will update automatically on new commits.