Several techniques that people use to establish literary dependence include:
- Identical passages of several words or more
- Unusual or unexpected words matching (especially usages that are idiosyncratic to one of the authors)
- Overal structure matching
- Matches in narrative or parenthetical material (this precludes the possibility that both are just quoting the same source)
- Editorial fatigue where the second author accidentally copies some material that's inconsistent with the editorial changes they made earlier.
A good introductory source for learning about some of this stuff is Goodacre's textbook on the Synoptic Problem.
Although I think it'd be unfair to call these techniques subjective, they are qualitative rather than quantitative. This is to be expected since the educational training of biblical scholars typically does not include a lot of heavy statistics and mathematics. However, in the computer age there are increasing attempts to apply quantitative techniques to literary questions. A remarkable recent example I'd heard of (which isn't directly relevant to the question of literary dependence) is Stephen Carlson's phylogenetic analysis of the textual history of Galatians.
There have been several attempts to quantify questions of literary dependence and direction of literary dependence. I think the earliest for the Gospels was in Tyson and Longstaff's Synoptic Abstract. But I can't seem to find much info on their findings. But even though there's been some attempts, it seems to me that there hasn't yet been a really conclusive and satisfying quantitative model built for addressing questions of literary dependence. Though I do wonder whether companies which detect plagiarism of essays are sitting on good algorithms for this sort of question.
In response to swasheck's question, let me try to spin out a bit more the difference between valid/invalid, objective/subjective, and quantitative/qualitative. Please take this with a grain of salt because I'm not an expert and may be saying things that are totally wrong.
Let's suppose for concreteness that we're looking at the following situation: an examiner has chosen a bunch of excerpts of newspaper articles, and has challenged various experts to detect which articles are dependent on which others. The examiner knows the answers (because she knows which ones came from wire services, and what times the articles were written, and has confirmed some of this information with the authors), but the experts do not know the answers. Each expert uses a different technique (possibly computer assisted) to determine their own answers, which are then given to the examiner.
Here "validity" is asking how often the expert gets the right answer (assuming we have a large enough sample to ignore luck). If an expert gets the answer right the vast majority of the time, then their technique is almost surely a valid way of getting right answers.
"Objectivity" or "subjectivity" is a different measure from validity. If one experts technique is just "I'm an expert and I know it when I see it," that could be valid but would also be highly subjective because a different person might have different intuitions. Nonetheless there could be a very successful and valid subjective technique.
"Quantitative" vs. "qualitative" is again a description of the kind of technique used. A quantitative measure involves counting things and doing calculations, while a "qualitative" one involves less numerical approaches. For example, it is probably quite difficult to turn "editorial fatigue" into a quantitative technique, because it involves looking at the meaning and not just the words used, while counting lengths of longest strings is quantitative. You can have a method that's "quantitative" but totally invalid because it doesn't measure the right things. Similarly you could have one that is qualitative but quite valid.
You can have techniques that are qualitative but nonetheless objective, if they're teachable to a person but difficult to program. Say you have some qualitative technique that takes a semester to teach to graduate students but once you've taught it to those students they will agree with each other 99% of the time. Then even though this technique is qualitative, nonetheless it's objective. Conversely, you could have a highly quantitative technique that just involved counting words, which nonetheless was quite subjective because say it didn't make it clear when to count two conjugations or declensions as the "same word."
In social science, a common technique of analyzing human behavior is to come up with a "coding system" where you replace the video of people with a collection of specific events fitting into some system. The goal of such a system is always to turn something qualitative into something quantitative. But just because it's quantitative doesn't mean it's objective or valid. You have to check that different coders using looking at the same video give the same outputs, and you have to be careful that you're measuring what you intended to measure rather than something similar.